[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:31:55 dist-test-slave-dxdd krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:31:55 dist-test-slave-dxdd krb5kdc[395](info): set up 2 sockets
Sep 25 10:31:55 dist-test-slave-dxdd krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 10:31:56.751157 388 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:31:56 dist-test-slave-dxdd krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:31:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:31:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:31:56.922510 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43181
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:31:57.244700 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:31:57.244978 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:31:57.245152 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:31:57.252167 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:31:57.252293 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:31:57.252439 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:31:57.252542 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:31:57.252625 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:31:57.263288 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:31:57.264684 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:31:57.266002 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:31:57.278154 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:57.278249 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:57.278227 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:31:57.278936 408 server_base.cc:1034] running on GCE node
I20260925 10:31:57.279780 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:31:57.281127 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:31:57.282413 408 hybrid_clock.cc:648] HybridClock initialized: now 1790332317282259 us; error 60 us; skew 500 ppm
I20260925 10:31:57.282749 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:31:57 dist-test-slave-dxdd krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:31:57.291436 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:31:57.292934 408 webserver.cc:460] Webserver started at http://127.0.97.62:45869/ using document root <none> and password file <none>
I20260925 10:31:57.293655 408 fs_manager.cc:359] Metadata directory not provided
I20260925 10:31:57.293787 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:31:57.294094 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:31:57.297190 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "ff7ff07611a74f8ab928c41515aaf044"
format_stamp: "Formatted at 2026-09-25 10:31:57 on dist-test-slave-dxdd"
server_key: "ded71e26f6bd0a15fa146825966bfbd6"
server_key_iv: "3425ccc569cabc12c0529c7859a1e95a"
server_key_version: "encryptionkey@0"
I20260925 10:31:57.297941 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "ff7ff07611a74f8ab928c41515aaf044"
format_stamp: "Formatted at 2026-09-25 10:31:57 on dist-test-slave-dxdd"
server_key: "ded71e26f6bd0a15fa146825966bfbd6"
server_key_iv: "3425ccc569cabc12c0529c7859a1e95a"
server_key_version: "encryptionkey@0"
I20260925 10:31:57.303556 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 10:31:57.306959 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:57.308238 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:31:57.308403 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "ff7ff07611a74f8ab928c41515aaf044"
format_stamp: "Formatted at 2026-09-25 10:31:57 on dist-test-slave-dxdd"
server_key: "ded71e26f6bd0a15fa146825966bfbd6"
server_key_iv: "3425ccc569cabc12c0529c7859a1e95a"
server_key_version: "encryptionkey@0"
I20260925 10:31:57.308609 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:31:57.328671 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:31:57.332408 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:31:57.332680 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:31:57.351606 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43181
I20260925 10:31:57.351635 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43181 every 8 connection(s)
I20260925 10:31:57.352903 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:31:57.356379 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:31:57.358880 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 408
I20260925 10:31:57.359192 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:31:57.361856 388 external_mini_cluster.cc:1428] Setting key f4fd340cdc97203fd03e420fbc41d1fc
I20260925 10:31:57.369442 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044: Bootstrap starting.
I20260925 10:31:57.372190 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044: Neither blocks nor log segments found. Creating new log.
I20260925 10:31:57.373068 475 log.cc:824] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044: Log is configured to *not* fsync() on all Append() calls
I20260925 10:31:57.375620 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044: No bootstrap required, opened a new log
I20260925 10:31:57.385298 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7ff07611a74f8ab928c41515aaf044" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43181 } }
I20260925 10:31:57.385725 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:31:57.385833 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff7ff07611a74f8ab928c41515aaf044, State: Initialized, Role: FOLLOWER
I20260925 10:31:57.386487 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7ff07611a74f8ab928c41515aaf044" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43181 } }
I20260925 10:31:57.386790 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:31:57.386940 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:31:57.387104 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:31:57.389377 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7ff07611a74f8ab928c41515aaf044" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43181 } }
I20260925 10:31:57.389840 475 leader_election.cc:302] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff7ff07611a74f8ab928c41515aaf044; no voters:
I20260925 10:31:57.390609 475 leader_election.cc:288] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:31:57.390697 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [term 1 FOLLOWER]: Leader election won for term 1
Sep 25 10:31:57 dist-test-slave-dxdd krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:31:57.391974 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [term 1 LEADER]: Becoming Leader. State: Replica: ff7ff07611a74f8ab928c41515aaf044, State: Running, Role: LEADER
I20260925 10:31:57.392828 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:31:57.392937 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7ff07611a74f8ab928c41515aaf044" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43181 } }
I20260925 10:31:57.401855 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:57.366158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56512 (local address 127.0.97.62:43181)
0925 10:31:57.367477 (+ 1319us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:57.367495 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:57.368642 (+ 1147us) server_negotiation.cc:407] Connection header received
0925 10:31:57.377869 (+ 9227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:57.377893 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:57.378359 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:57.378750 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:57.381122 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:57.383624 (+ 2502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:57.385273 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:57.386294 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:57.386485 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:57.392958 (+ 6473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:57.392973 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:57.392994 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:57.393049 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:57.396815 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:57.397624 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:57.397632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:57.397646 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:57.397812 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:57.398261 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:57.398268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:57.398271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:57.398974 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:57.399246 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:57.400143 (+ 897us) server_negotiation.cc:299] Negotiation successful
0925 10:31:57.400525 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1066,"thread_start_us":377,"threads_started":1}
I20260925 10:31:57.404631 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff7ff07611a74f8ab928c41515aaf044. Latest consensus state: current_term: 1 leader_uuid: "ff7ff07611a74f8ab928c41515aaf044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7ff07611a74f8ab928c41515aaf044" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43181 } } }
I20260925 10:31:57.405087 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:57.408757 495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:31:57.408675 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff7ff07611a74f8ab928c41515aaf044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7ff07611a74f8ab928c41515aaf044" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43181 } } }
I20260925 10:31:57.408998 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044 [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:57.414453 495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:31:57.422358 495 catalog_manager.cc:1347] Generated new cluster ID: b2a1edf18c9b4a789935eadfa8ad545e
I20260925 10:31:57.422472 495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:31:57.436388 495 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:31:57.437556 495 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:31:57.445426 495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff7ff07611a74f8ab928c41515aaf044: Generated new TSK 0
I20260925 10:31:57.446197 495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:31:57.495955 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43181
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:31:57.852255 502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:31:57.852521 502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:31:57.852615 502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:31:57.859578 502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:31:57.859674 502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:31:57.859788 502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:31:57.869541 502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43181
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:31:57.870707 502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:31:57.871989 502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:31:57.885459 507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:57.885428 510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:57.885509 508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:31:57.886433 502 server_base.cc:1034] running on GCE node
I20260925 10:31:57.887003 502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:31:57.887887 502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:31:57.889083 502 hybrid_clock.cc:648] HybridClock initialized: now 1790332317889053 us; error 30 us; skew 500 ppm
I20260925 10:31:57.889379 502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:31:57 dist-test-slave-dxdd krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:31:57.892375 502 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:31:57.893751 502 webserver.cc:460] Webserver started at http://127.0.97.1:41175/ using document root <none> and password file <none>
I20260925 10:31:57.894482 502 fs_manager.cc:359] Metadata directory not provided
I20260925 10:31:57.894585 502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:31:57.894831 502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:31:57.897526 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "5ca3ceeaf19f4996b6e6404c7dd71b70"
format_stamp: "Formatted at 2026-09-25 10:31:57 on dist-test-slave-dxdd"
server_key: "0be3f53081b2f82977054728770fe9bc"
server_key_iv: "5027f8ee0264951b17d51b82d4cd8d9d"
server_key_version: "encryptionkey@0"
I20260925 10:31:57.898270 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5ca3ceeaf19f4996b6e6404c7dd71b70"
format_stamp: "Formatted at 2026-09-25 10:31:57 on dist-test-slave-dxdd"
server_key: "0be3f53081b2f82977054728770fe9bc"
server_key_iv: "5027f8ee0264951b17d51b82d4cd8d9d"
server_key_version: "encryptionkey@0"
I20260925 10:31:57.903311 502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 10:31:57.906949 517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:57.908288 502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:31:57.908442 502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "5ca3ceeaf19f4996b6e6404c7dd71b70"
format_stamp: "Formatted at 2026-09-25 10:31:57 on dist-test-slave-dxdd"
server_key: "0be3f53081b2f82977054728770fe9bc"
server_key_iv: "5027f8ee0264951b17d51b82d4cd8d9d"
server_key_version: "encryptionkey@0"
I20260925 10:31:57.908635 502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:31:57.925675 502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:31:57.929303 502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:31:57.929605 502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:31:57.930596 502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:31:57.931838 502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:31:57.931906 502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:57.932030 502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:31:57.932094 502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:57.966068 502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36317
I20260925 10:31:57.966120 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36317 every 8 connection(s)
I20260925 10:31:57.967316 502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:31:57.973491 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 502
I20260925 10:31:57.973712 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:31:57.974296 388 external_mini_cluster.cc:1428] Setting key 21c9df1aab98d2035d2f6d025d25c396
Sep 25 10:31:57 dist-test-slave-dxdd krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:31:57.987072 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:57.970142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45441 (local address 127.0.97.62:43181)
0925 10:31:57.970588 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:57.970595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:57.971551 (+ 956us) server_negotiation.cc:407] Connection header received
0925 10:31:57.973251 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:57.973256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:57.973354 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:57.973471 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:57.975158 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:57.975988 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:57.977542 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:57.977939 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:57.978106 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:57.981666 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:57.981671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:57.981675 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:57.981704 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:57.984151 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:57.984860 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:57.984868 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:57.984876 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:57.984982 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:57.985396 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:57.985401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:57.985404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:57.985654 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:57.985806 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:57.986489 (+ 683us) server_negotiation.cc:299] Negotiation successful
0925 10:31:57.986696 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":237,"threads_started":1}
I20260925 10:31:57.987978 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:57.970748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43181 (local address 127.0.97.1:45441)
0925 10:31:57.971392 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:57.971428 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:57.972967 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:57.973739 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:57.973747 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:57.974276 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:57.974947 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:57.974970 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:57.976158 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:57.976163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:57.977390 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:57.977401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:57.978208 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:57.978215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:57.978497 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:57.979184 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:57.979209 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:57.981459 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:57.984291 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:57.984299 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:57.984315 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:57.984734 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:57.985102 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:57.985108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:57.985111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:57.985281 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:57.985776 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:57.985784 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:57.986096 (+ 312us) client_negotiation.cc:769] Sending connection context
0925 10:31:57.986384 (+ 288us) client_negotiation.cc:240] Negotiation successful
0925 10:31:57.986875 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":250,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260925 10:31:57.989470 630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43181
I20260925 10:31:57.989759 630 heartbeater.cc:464] Registering TS with master...
I20260925 10:31:57.990561 630 heartbeater.cc:511] Master 127.0.97.62:43181 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260925 10:31:57.992612 440 ts_manager.cc:194] Registered new tserver with Master: 5ca3ceeaf19f4996b6e6404c7dd71b70 (127.0.97.1:36317)
I20260925 10:31:57.994131 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:31:58.013299 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43181
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:31:58.335569 636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:31:58.335835 636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:31:58.335927 636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:31:58.342842 636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:31:58.342941 636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:31:58.343050 636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:31:58.353343 636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43181
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:31:58.354733 636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:31:58.356271 636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:31:58.371446 642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:58.372467 644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:58.372699 641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:31:58.373126 636 server_base.cc:1034] running on GCE node
I20260925 10:31:58.373761 636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:31:58.374888 636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:31:58.376071 636 hybrid_clock.cc:648] HybridClock initialized: now 1790332318375986 us; error 76 us; skew 500 ppm
I20260925 10:31:58.376470 636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:31:58 dist-test-slave-dxdd krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:31:58.380445 636 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:31:58.381907 636 webserver.cc:460] Webserver started at http://127.0.97.2:37745/ using document root <none> and password file <none>
I20260925 10:31:58.382719 636 fs_manager.cc:359] Metadata directory not provided
I20260925 10:31:58.382838 636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:31:58.383178 636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:31:58.385792 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "40bacacbf8954de29b6881a97e01f15e"
format_stamp: "Formatted at 2026-09-25 10:31:58 on dist-test-slave-dxdd"
server_key: "15e876453907f67071c1bc323b0f1e05"
server_key_iv: "c6ad6a35cae0d992145552ba6423d237"
server_key_version: "encryptionkey@0"
I20260925 10:31:58.386632 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "40bacacbf8954de29b6881a97e01f15e"
format_stamp: "Formatted at 2026-09-25 10:31:58 on dist-test-slave-dxdd"
server_key: "15e876453907f67071c1bc323b0f1e05"
server_key_iv: "c6ad6a35cae0d992145552ba6423d237"
server_key_version: "encryptionkey@0"
I20260925 10:31:58.391710 636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 10:31:58.394979 651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:58.396159 636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 10:31:58.396276 636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "40bacacbf8954de29b6881a97e01f15e"
format_stamp: "Formatted at 2026-09-25 10:31:58 on dist-test-slave-dxdd"
server_key: "15e876453907f67071c1bc323b0f1e05"
server_key_iv: "c6ad6a35cae0d992145552ba6423d237"
server_key_version: "encryptionkey@0"
I20260925 10:31:58.396417 636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:31:58.408419 636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:31:58.412182 636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:31:58.412463 636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:31:58.413817 636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:31:58.415047 636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:31:58.415129 636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:58.415201 636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:31:58.415243 636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:58.447815 636 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43575
I20260925 10:31:58.447852 763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43575 every 8 connection(s)
I20260925 10:31:58.449018 636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:31:58.449638 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 636
I20260925 10:31:58.450217 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:31:58.450780 388 external_mini_cluster.cc:1428] Setting key 3fc25c6f132ddc5a5beb96181125342f
Sep 25 10:31:58 dist-test-slave-dxdd krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260925 10:31:58.469223 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:58.452121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46681 (local address 127.0.97.62:43181)
0925 10:31:58.452319 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:58.452325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:58.453695 (+ 1370us) server_negotiation.cc:407] Connection header received
0925 10:31:58.455568 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:58.455574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:58.455660 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:58.455773 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:58.457567 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:58.458254 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:58.459755 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:58.460178 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:58.460309 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:58.463549 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:58.463558 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:58.463562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:58.463590 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:58.466361 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:58.467028 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:58.467034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:58.467038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:58.467158 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:58.467608 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:58.467615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:58.467618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:58.467861 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:58.468003 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:58.468689 (+ 686us) server_negotiation.cc:299] Negotiation successful
0925 10:31:58.468895 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:31:58.470196 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:58.452831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43181 (local address 127.0.97.2:46681)
0925 10:31:58.453548 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:58.453593 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:58.455304 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:58.455991 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:58.455999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:58.456488 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:58.457395 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:58.457412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:58.458464 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:58.458468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:58.459615 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:58.459625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:58.460390 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:58.460396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:58.460607 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:58.461277 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:58.461297 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:58.463350 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:58.466489 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:58.466498 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:58.466511 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:58.466910 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:58.467297 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:58.467302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:58.467305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:58.467497 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:58.467994 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:58.468001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:58.468353 (+ 352us) client_negotiation.cc:769] Sending connection context
0925 10:31:58.468603 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 10:31:58.469092 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":307,"threads_started":1}
I20260925 10:31:58.471613 764 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43181
I20260925 10:31:58.471933 764 heartbeater.cc:464] Registering TS with master...
I20260925 10:31:58.472635 764 heartbeater.cc:511] Master 127.0.97.62:43181 requested a full tablet report, sending...
I20260925 10:31:58.473971 440 ts_manager.cc:194] Registered new tserver with Master: 40bacacbf8954de29b6881a97e01f15e (127.0.97.2:43575)
I20260925 10:31:58.474777 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:31:58.489178 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43181
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:31:58.804949 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:31:58.805286 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:31:58.805460 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:31:58.812989 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:31:58.813125 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:31:58.813297 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:31:58.822885 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43181
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:31:58.824149 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:31:58.825459 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:31:58.839641 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:58.839701 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:58.839709 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:31:58.840484 769 server_base.cc:1034] running on GCE node
I20260925 10:31:58.841065 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:31:58.842059 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:31:58.843309 769 hybrid_clock.cc:648] HybridClock initialized: now 1790332318843222 us; error 73 us; skew 500 ppm
I20260925 10:31:58.843629 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:31:58 dist-test-slave-dxdd krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:31:58.846606 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:31:58.847895 769 webserver.cc:460] Webserver started at http://127.0.97.3:35925/ using document root <none> and password file <none>
I20260925 10:31:58.848896 769 fs_manager.cc:359] Metadata directory not provided
I20260925 10:31:58.848992 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:31:58.849244 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:31:58.851980 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "4ddec6a81f34481cade05904c95fe202"
format_stamp: "Formatted at 2026-09-25 10:31:58 on dist-test-slave-dxdd"
server_key: "8b221f7d620b913422aaee1ac45b819a"
server_key_iv: "791709ea87ccd44e0cf959e6b4b22767"
server_key_version: "encryptionkey@0"
I20260925 10:31:58.852663 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4ddec6a81f34481cade05904c95fe202"
format_stamp: "Formatted at 2026-09-25 10:31:58 on dist-test-slave-dxdd"
server_key: "8b221f7d620b913422aaee1ac45b819a"
server_key_iv: "791709ea87ccd44e0cf959e6b4b22767"
server_key_version: "encryptionkey@0"
I20260925 10:31:58.857295 769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 10:31:58.860347 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:58.861538 769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:31:58.861686 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "4ddec6a81f34481cade05904c95fe202"
format_stamp: "Formatted at 2026-09-25 10:31:58 on dist-test-slave-dxdd"
server_key: "8b221f7d620b913422aaee1ac45b819a"
server_key_iv: "791709ea87ccd44e0cf959e6b4b22767"
server_key_version: "encryptionkey@0"
I20260925 10:31:58.861847 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:31:58.877332 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:31:58.881421 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:31:58.881690 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:31:58.882622 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:31:58.883831 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:31:58.883900 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:58.884070 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:31:58.884141 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:58.918552 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40555
I20260925 10:31:58.918583 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40555 every 8 connection(s)
I20260925 10:31:58.919808 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:31:58.922817 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 769
I20260925 10:31:58.922989 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:31:58.923543 388 external_mini_cluster.cc:1428] Setting key a10835574821bb1e0880c430ee71abb0
Sep 25 10:31:58 dist-test-slave-dxdd krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:31:58.940153 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:58.922230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55027 (local address 127.0.97.62:43181)
0925 10:31:58.922451 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:58.922459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:58.923867 (+ 1408us) server_negotiation.cc:407] Connection header received
0925 10:31:58.926848 (+ 2981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:58.926853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:58.926940 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:58.927040 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:58.928493 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:58.929245 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:58.930940 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:58.931315 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:58.931445 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:58.934693 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:58.934698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:58.934701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:58.934736 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:58.937215 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:58.937997 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:58.938009 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:58.938011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:58.938130 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:58.938664 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:58.938668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:58.938671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:58.938967 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:58.939125 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:58.939778 (+ 653us) server_negotiation.cc:299] Negotiation successful
0925 10:31:58.939922 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 10:31:58.941039 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:58.922796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43181 (local address 127.0.97.3:55027)
0925 10:31:58.923680 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:58.923739 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:58.926550 (+ 2811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:58.927280 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:58.927288 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:58.927783 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:58.928335 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:58.928351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:58.929405 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:58.929410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:58.930807 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:58.930818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:58.931518 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:58.931527 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:58.931766 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:58.932429 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:58.932451 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:58.934495 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:58.937348 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:58.937358 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:58.937373 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:58.937873 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:58.938255 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:58.938258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:58.938260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:58.938537 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:58.939117 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:58.939124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:58.939456 (+ 332us) client_negotiation.cc:769] Sending connection context
0925 10:31:58.939669 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 10:31:58.940009 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":340,"threads_started":1}
I20260925 10:31:58.942330 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43181
I20260925 10:31:58.942577 897 heartbeater.cc:464] Registering TS with master...
I20260925 10:31:58.943203 897 heartbeater.cc:511] Master 127.0.97.62:43181 requested a full tablet report, sending...
I20260925 10:31:58.944446 440 ts_manager.cc:194] Registered new tserver with Master: 4ddec6a81f34481cade05904c95fe202 (127.0.97.3:40555)
I20260925 10:31:58.945283 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55027
I20260925 10:31:58.951272 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:31:58 dist-test-slave-dxdd krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:31:58 dist-test-slave-dxdd krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:31:58.993259 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:58.979376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.0.97.1:36317)
0925 10:31:58.980100 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:58.980107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:58.980132 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:31:58.980192 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:58.980198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:58.980377 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:58.980498 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:58.981771 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:58.982814 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:58.983974 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:58.984470 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:58.984572 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:58.988116 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:58.988125 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:58.988137 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:58.988165 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:58.990619 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:58.991610 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:58.991616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:58.991619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:58.991751 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:58.992063 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:58.992066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:58.992068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:58.992366 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:58.992562 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:58.992767 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 10:31:58.992990 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":476,"threads_started":1}
W20260925 10:31:58.994482 545 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52456
I20260925 10:31:58.996758 630 heartbeater.cc:503] Master 127.0.97.62:43181 was elected leader, sending a full tablet report...
Sep 25 10:31:59 dist-test-slave-dxdd krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:31:59 dist-test-slave-dxdd krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:31:59.035856 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:59.023011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52468 (local address 127.0.97.1:36317)
0925 10:31:59.023221 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:59.023230 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:59.023387 (+ 157us) server_negotiation.cc:407] Connection header received
0925 10:31:59.023532 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:59.023537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:59.023601 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:59.023684 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:59.024919 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:59.025592 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:59.026809 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:59.027221 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:59.027311 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:59.031015 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:59.031020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:59.031023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:59.031054 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:59.033428 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:59.034026 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:59.034033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:59.034036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:59.034152 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:59.034560 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:59.034567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:59.034570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:59.034813 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:59.035038 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:59.035337 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 10:31:59.035536 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 10:31:59.037314 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 502
I20260925 10:31:59.050007 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:59.038149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.0.97.1:36317)
0925 10:31:59.038296 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:59.038299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:59.038685 (+ 386us) server_negotiation.cc:407] Connection header received
0925 10:31:59.038800 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:59.038803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:59.038851 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:59.038935 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:59.039948 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:59.040577 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:59.041657 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:59.042022 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:59.042101 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:59.044530 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:59.044534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:59.044537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:59.044572 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:59.047047 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:59.047863 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:59.047870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:59.047875 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:59.048078 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:59.048464 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:59.048469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:59.048472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:59.048831 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:59.049099 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:59.049390 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 10:31:59.049666 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:31:59.050863 625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:31:59.134479 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 502
I20260925 10:31:59.150394 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 636
Sep 25 10:31:59 dist-test-slave-dxdd krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:31:59.166018 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:59.151224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.0.97.2:43575)
0925 10:31:59.151723 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:59.151730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:59.151755 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:31:59.151812 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:59.151818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:59.152026 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:59.152155 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:59.153188 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:59.154185 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:59.155373 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:59.156361 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:59.156500 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:59.160752 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:59.160763 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:59.160777 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:59.160810 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:59.163691 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:59.164251 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:59.164258 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:59.164260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:59.164373 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:59.164712 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:59.164715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:59.164717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:59.165038 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:59.165247 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:59.165488 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 10:31:59.165678 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":262,"threads_started":1}
I20260925 10:31:59.167466 759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:31:59.248597 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 636
I20260925 10:31:59.271277 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 769
Sep 25 10:31:59 dist-test-slave-dxdd krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:31:59.287315 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:59.272170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38546 (local address 127.0.97.3:40555)
0925 10:31:59.272704 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:59.272711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:59.272738 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:31:59.272793 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:59.272798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:59.273006 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:59.273175 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:59.274432 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:59.275545 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:59.277703 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:59.278233 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:59.278420 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:59.282506 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:59.282516 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:59.282530 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:59.282561 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:59.285125 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:59.285660 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:59.285682 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:59.285684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:59.285786 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:59.286077 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:59.286080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:59.286082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:59.286434 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:59.286647 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:59.286877 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 10:31:59.287054 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":264,"threads_started":1}
I20260925 10:31:59.288645 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:31:59.373327 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 769
I20260925 10:31:59.389670 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 408
I20260925 10:31:59.390859 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:31:59.492847 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 408
2026-09-25T10:31:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3822 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:32:01 dist-test-slave-dxdd krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:32:01 dist-test-slave-dxdd krb5kdc[918](info): set up 2 sockets
Sep 25 10:32:01 dist-test-slave-dxdd krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260925 10:32:02.094457 388 mini_kdc.cc:121] Time spent starting KDC: real 2.561s user 0.009s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 10:32:02.256441 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.162s user 0.001s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 10:32:02.372406 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.116s user 0.001s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 10:32:02.556502 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.184s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/test-user.keytab.
W20260925 10:32:03.034125 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.477s user 0.000s sys 0.004s
Sep 25 10:32:03 dist-test-slave-dxdd krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:32:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:32:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
W20260925 10:32:03.590600 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.62: real 0.512s user 0.000s sys 0.007s
I20260925 10:32:03.591616 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35835
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:03.903179 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:03.903441 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:03.903527 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:03.910399 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:03.910495 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:03.910547 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:03.910601 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:03.910643 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:32:03.919836 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:03.920970 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:03.922210 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:03.934134 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:03.934132 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:03.934168 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:03.934847 931 server_base.cc:1034] running on GCE node
I20260925 10:32:03.935684 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:03.936930 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:03.938153 931 hybrid_clock.cc:648] HybridClock initialized: now 1790332323938113 us; error 54 us; skew 500 ppm
I20260925 10:32:03.938552 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:03 dist-test-slave-dxdd krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:03.945271 931 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:32:03.946796 931 webserver.cc:460] Webserver started at http://127.0.97.62:36737/ using document root <none> and password file <none>
I20260925 10:32:03.947419 931 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:03.947513 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:03.947770 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:03.951195 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "f35696a36e21458bb090475c913b16cb"
format_stamp: "Formatted at 2026-09-25 10:32:03 on dist-test-slave-dxdd"
server_key: "a8d679ebe6caa8c3cfbab1e58a7ff3ec"
server_key_iv: "20fb61b4ed416c6ae7cebc7c1b6938eb"
server_key_version: "encryptionkey@0"
I20260925 10:32:03.951934 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "f35696a36e21458bb090475c913b16cb"
format_stamp: "Formatted at 2026-09-25 10:32:03 on dist-test-slave-dxdd"
server_key: "a8d679ebe6caa8c3cfbab1e58a7ff3ec"
server_key_iv: "20fb61b4ed416c6ae7cebc7c1b6938eb"
server_key_version: "encryptionkey@0"
I20260925 10:32:03.956918 931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:32:03.960238 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:03.961439 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:32:03.961622 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "f35696a36e21458bb090475c913b16cb"
format_stamp: "Formatted at 2026-09-25 10:32:03 on dist-test-slave-dxdd"
server_key: "a8d679ebe6caa8c3cfbab1e58a7ff3ec"
server_key_iv: "20fb61b4ed416c6ae7cebc7c1b6938eb"
server_key_version: "encryptionkey@0"
I20260925 10:32:03.961799 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:03.979547 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:03.983788 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:03.984063 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:04.001833 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35835
I20260925 10:32:04.001899 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35835 every 8 connection(s)
I20260925 10:32:04.003217 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:32:04.006019 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 931
I20260925 10:32:04.006204 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:32:04.006815 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:04.006876 388 external_mini_cluster.cc:1428] Setting key 82fc53c1cce082e9e5909bcfa055d9c6
I20260925 10:32:04.020424 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb: Bootstrap starting.
Sep 25 10:32:04 dist-test-slave-dxdd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:04.024132 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:04.025419 998 log.cc:824] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:04.028745 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb: No bootstrap required, opened a new log
I20260925 10:32:04.030407 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:04.008792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44442 (local address 127.0.97.62:35835)
0925 10:32:04.009597 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:04.009613 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:04.009665 (+ 52us) server_negotiation.cc:407] Connection header received
0925 10:32:04.011569 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:04.011596 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:04.012077 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:04.012501 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:04.013676 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:04.015664 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:04.016884 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:04.017965 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:04.018155 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:04.021957 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:04.021968 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:04.021986 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:04.022043 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:04.025975 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:04.026838 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:04.026848 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:04.026855 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:04.027039 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:04.027364 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:04.027369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:04.027372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:04.028043 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:04.028325 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:04.028682 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 10:32:04.029105 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":331,"threads_started":1}
I20260925 10:32:04.036723 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f35696a36e21458bb090475c913b16cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35835 } }
I20260925 10:32:04.037089 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:04.037211 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f35696a36e21458bb090475c913b16cb, State: Initialized, Role: FOLLOWER
I20260925 10:32:04.037732 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f35696a36e21458bb090475c913b16cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35835 } }
I20260925 10:32:04.037962 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:32:04.038097 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:32:04.038264 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:04.054461 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f35696a36e21458bb090475c913b16cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35835 } }
I20260925 10:32:04.055027 998 leader_election.cc:302] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f35696a36e21458bb090475c913b16cb; no voters:
I20260925 10:32:04.055629 998 leader_election.cc:288] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:32:04.055764 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:04.057986 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [term 1 LEADER]: Becoming Leader. State: Replica: f35696a36e21458bb090475c913b16cb, State: Running, Role: LEADER
I20260925 10:32:04.058609 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f35696a36e21458bb090475c913b16cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35835 } }
I20260925 10:32:04.058969 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:04.061697 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f35696a36e21458bb090475c913b16cb. Latest consensus state: current_term: 1 leader_uuid: "f35696a36e21458bb090475c913b16cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f35696a36e21458bb090475c913b16cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35835 } } }
I20260925 10:32:04.062122 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:04.062501 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f35696a36e21458bb090475c913b16cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f35696a36e21458bb090475c913b16cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35835 } } }
I20260925 10:32:04.062748 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:04.067276 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:04.071307 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:04.079209 1012 catalog_manager.cc:1347] Generated new cluster ID: 65d305a5e1704b4ba9abe597ceb3cb52
I20260925 10:32:04.079319 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:04.086410 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:32:04.087580 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:04.095893 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f35696a36e21458bb090475c913b16cb: Generated new TSK 0
I20260925 10:32:04.096629 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
W20260925 10:32:04.743136 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.1: real 0.632s user 0.002s sys 0.007s
I20260925 10:32:04.744050 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35835
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:05.051647 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:05.051939 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:05.052093 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:05.059055 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:05.059185 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:05.059361 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:32:05.068825 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35835
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:05.069926 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:05.071156 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:05.083622 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:05.083670 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:05.083590 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:05.084355 1024 server_base.cc:1034] running on GCE node
I20260925 10:32:05.085003 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:05.085902 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:05.087118 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790332325087078 us; error 54 us; skew 500 ppm
I20260925 10:32:05.087417 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:05 dist-test-slave-dxdd krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:05.090423 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:32:05.091646 1024 webserver.cc:460] Webserver started at http://127.0.97.1:37707/ using document root <none> and password file <none>
I20260925 10:32:05.092239 1024 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:05.092332 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:05.092576 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:05.095261 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "27e79e60a954482ea9b895036fed3040"
format_stamp: "Formatted at 2026-09-25 10:32:05 on dist-test-slave-dxdd"
server_key: "c3b0d8f534d2cd494bc1dfad036fec75"
server_key_iv: "d6dd387cf3f97c8697a406340b419277"
server_key_version: "encryptionkey@0"
I20260925 10:32:05.095999 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "27e79e60a954482ea9b895036fed3040"
format_stamp: "Formatted at 2026-09-25 10:32:05 on dist-test-slave-dxdd"
server_key: "c3b0d8f534d2cd494bc1dfad036fec75"
server_key_iv: "d6dd387cf3f97c8697a406340b419277"
server_key_version: "encryptionkey@0"
I20260925 10:32:05.100734 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:32:05.103922 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:05.105154 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:32:05.105319 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "27e79e60a954482ea9b895036fed3040"
format_stamp: "Formatted at 2026-09-25 10:32:05 on dist-test-slave-dxdd"
server_key: "c3b0d8f534d2cd494bc1dfad036fec75"
server_key_iv: "d6dd387cf3f97c8697a406340b419277"
server_key_version: "encryptionkey@0"
I20260925 10:32:05.105556 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:05.119964 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:05.123961 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:05.124230 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:05.125049 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:05.126556 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:05.126655 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:05.126768 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:05.126830 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:05.157806 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44663
I20260925 10:32:05.157883 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44663 every 8 connection(s)
I20260925 10:32:05.159040 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:32:05.168990 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 1024
I20260925 10:32:05.169214 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:32:05.169955 388 external_mini_cluster.cc:1428] Setting key e99af2df1ef8e76361ebf5872945c65f
Sep 25 10:32:05 dist-test-slave-dxdd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:05.178438 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:05.161438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50983 (local address 127.0.97.62:35835)
0925 10:32:05.161900 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:05.161906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:05.163026 (+ 1120us) server_negotiation.cc:407] Connection header received
0925 10:32:05.164669 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:05.164673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:05.164731 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:05.164822 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:05.166482 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:05.167201 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:05.168618 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:05.168995 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:05.169129 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:05.172978 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:05.172994 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:05.173002 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:05.173031 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:05.175564 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:05.176237 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:05.176242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:05.176250 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:05.176362 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:05.176780 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:05.176785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:05.176788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:05.177034 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:05.177213 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:05.177846 (+ 633us) server_negotiation.cc:299] Negotiation successful
0925 10:32:05.178059 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":197,"threads_started":1}
I20260925 10:32:05.179299 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:05.162095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35835 (local address 127.0.97.1:50983)
0925 10:32:05.162801 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:05.162856 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:05.164449 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:05.165043 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:05.165050 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:05.165602 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:05.166219 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:05.166237 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:05.167318 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:05.167321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:05.168502 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:05.168512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:05.169212 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:05.169219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:05.169514 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:05.170233 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:05.170253 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:05.172765 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:05.175703 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:05.175711 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:05.175726 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:05.176114 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:05.176467 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:05.176473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:05.176476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:05.176665 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:05.177177 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:05.177186 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:05.177492 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 10:32:05.177745 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 10:32:05.178230 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":318,"threads_started":1}
I20260925 10:32:05.180804 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35835
I20260925 10:32:05.181134 1152 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:05.182017 1152 heartbeater.cc:511] Master 127.0.97.62:35835 requested a full tablet report, sending...
I20260925 10:32:05.184440 963 ts_manager.cc:194] Registered new tserver with Master: 27e79e60a954482ea9b895036fed3040 (127.0.97.1:44663)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260925 10:32:05.186390 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:32:05.207468 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35835
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:05.507448 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:05.507697 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:05.507787 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:05.514734 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:05.514824 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:05.514937 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:32:05.523864 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35835
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:05.524883 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:05.526103 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:05.538229 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:05.538187 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:05.538267 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:05.538935 1158 server_base.cc:1034] running on GCE node
I20260925 10:32:05.539507 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:05.540335 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:05.541550 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790332325541500 us; error 81 us; skew 500 ppm
I20260925 10:32:05.541834 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:05 dist-test-slave-dxdd krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:05.544793 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:32:05.546077 1158 webserver.cc:460] Webserver started at http://127.0.97.2:42049/ using document root <none> and password file <none>
I20260925 10:32:05.546768 1158 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:05.546905 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:05.547214 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:05.549935 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "d91bd616b36b46c4986c32c107720d9b"
format_stamp: "Formatted at 2026-09-25 10:32:05 on dist-test-slave-dxdd"
server_key: "910f646b052e5326c497d989fc6434d8"
server_key_iv: "947be2c3ae5349ccbc9e33731c3ed3cb"
server_key_version: "encryptionkey@0"
I20260925 10:32:05.550673 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d91bd616b36b46c4986c32c107720d9b"
format_stamp: "Formatted at 2026-09-25 10:32:05 on dist-test-slave-dxdd"
server_key: "910f646b052e5326c497d989fc6434d8"
server_key_iv: "947be2c3ae5349ccbc9e33731c3ed3cb"
server_key_version: "encryptionkey@0"
I20260925 10:32:05.555270 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:32:05.558283 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:05.559626 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:32:05.559778 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "d91bd616b36b46c4986c32c107720d9b"
format_stamp: "Formatted at 2026-09-25 10:32:05 on dist-test-slave-dxdd"
server_key: "910f646b052e5326c497d989fc6434d8"
server_key_iv: "947be2c3ae5349ccbc9e33731c3ed3cb"
server_key_version: "encryptionkey@0"
I20260925 10:32:05.559954 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:05.580811 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:05.584831 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:05.585112 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:05.586077 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:05.587777 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:05.587875 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:05.587998 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:05.588056 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:05.619524 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38525
I20260925 10:32:05.619606 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38525 every 8 connection(s)
I20260925 10:32:05.620806 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:32:05.621845 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 1158
I20260925 10:32:05.622038 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:32:05.622608 388 external_mini_cluster.cc:1428] Setting key bb254e412f04790ceebdf3a3d64e1ef2
Sep 25 10:32:05 dist-test-slave-dxdd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260925 10:32:05.641722 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:05.623753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34783 (local address 127.0.97.62:35835)
0925 10:32:05.623922 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:05.623926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:05.625364 (+ 1438us) server_negotiation.cc:407] Connection header received
0925 10:32:05.627213 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:05.627221 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:05.627309 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:05.627417 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:05.628760 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:05.629940 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:05.631466 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:05.632240 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:05.632381 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:05.636088 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:05.636092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:05.636095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:05.636127 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:05.638685 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:05.639395 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:05.639398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:05.639401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:05.639509 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:05.639845 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:05.639848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:05.639850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:05.640178 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:05.640285 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:05.641043 (+ 758us) server_negotiation.cc:299] Negotiation successful
0925 10:32:05.641265 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:32:05.642439 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:05.624411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35835 (local address 127.0.97.2:34783)
0925 10:32:05.625201 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:05.625250 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:05.626943 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:05.627592 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:05.627600 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:05.628070 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:05.628597 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:05.628615 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:05.630064 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:05.630067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:05.631334 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:05.631345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:05.632423 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:05.632427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:05.632599 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:05.633232 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:05.633252 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:05.635878 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:05.638830 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:05.638836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:05.638847 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:05.639274 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:05.639597 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:05.639600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:05.639602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:05.639753 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:05.640277 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:05.640282 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:05.640637 (+ 355us) client_negotiation.cc:769] Sending connection context
0925 10:32:05.640925 (+ 288us) client_negotiation.cc:240] Negotiation successful
0925 10:32:05.641265 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":349,"threads_started":1}
I20260925 10:32:05.643837 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35835
I20260925 10:32:05.644124 1286 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:05.644776 1286 heartbeater.cc:511] Master 127.0.97.62:35835 requested a full tablet report, sending...
I20260925 10:32:05.646209 963 ts_manager.cc:194] Registered new tserver with Master: d91bd616b36b46c4986c32c107720d9b (127.0.97.2:38525)
I20260925 10:32:05.647013 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:32:05.661273 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35835
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:05.965209 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:05.965607 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:05.965768 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:05.972867 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:05.973011 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:05.973210 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:32:05.982733 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35835
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:05.983942 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:05.985270 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:05.997998 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:05.997920 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:05.997964 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:05.998935 1291 server_base.cc:1034] running on GCE node
I20260925 10:32:05.999456 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:06.000339 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:06.001546 1291 hybrid_clock.cc:648] HybridClock initialized: now 1790332326001471 us; error 69 us; skew 500 ppm
I20260925 10:32:06.001879 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:05 dist-test-slave-dxdd krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:06.005098 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:32:06.006632 1291 webserver.cc:460] Webserver started at http://127.0.97.3:42419/ using document root <none> and password file <none>
I20260925 10:32:06.007272 1291 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:06.007418 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:06.007710 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:06.010675 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "9d2d211d2e18495aa7098b3b15011f22"
format_stamp: "Formatted at 2026-09-25 10:32:06 on dist-test-slave-dxdd"
server_key: "2fd408f3dcbc86c8e72e4241ff12f294"
server_key_iv: "0bdcde421547dc1c137d2af4cc07e306"
server_key_version: "encryptionkey@0"
I20260925 10:32:06.011408 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9d2d211d2e18495aa7098b3b15011f22"
format_stamp: "Formatted at 2026-09-25 10:32:06 on dist-test-slave-dxdd"
server_key: "2fd408f3dcbc86c8e72e4241ff12f294"
server_key_iv: "0bdcde421547dc1c137d2af4cc07e306"
server_key_version: "encryptionkey@0"
I20260925 10:32:06.016193 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:32:06.019410 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:06.020643 1291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:32:06.020813 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "9d2d211d2e18495aa7098b3b15011f22"
format_stamp: "Formatted at 2026-09-25 10:32:06 on dist-test-slave-dxdd"
server_key: "2fd408f3dcbc86c8e72e4241ff12f294"
server_key_iv: "0bdcde421547dc1c137d2af4cc07e306"
server_key_version: "encryptionkey@0"
I20260925 10:32:06.021005 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:06.063104 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:06.067682 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:06.067978 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:06.068934 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:06.070106 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:06.070181 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:06.070266 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:06.070322 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:06.102728 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44807
I20260925 10:32:06.102777 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44807 every 8 connection(s)
I20260925 10:32:06.103955 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:32:06.106371 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 1291
I20260925 10:32:06.106559 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:32:06.106967 388 external_mini_cluster.cc:1428] Setting key 05fe22d9f696ace2cd04686bd538d8be
Sep 25 10:32:06 dist-test-slave-dxdd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:06.123657 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.106469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43833 (local address 127.0.97.62:35835)
0925 10:32:06.107379 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:06.107383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:06.107977 (+ 594us) server_negotiation.cc:407] Connection header received
0925 10:32:06.110287 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:06.110291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:06.110406 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:06.110485 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:06.111806 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.112509 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.113917 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.114270 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.114455 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.117995 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:06.117999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:06.118002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:06.118032 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:06.120827 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:06.121597 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:06.121602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:06.121605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:06.121762 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:06.122241 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:06.122246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:06.122249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:06.122570 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:06.122666 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:06.123282 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 10:32:06.123428 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802}
I20260925 10:32:06.124719 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.106952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35835 (local address 127.0.97.3:43833)
0925 10:32:06.107793 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:06.107843 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:06.110029 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:06.110689 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:06.110697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:06.111163 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:06.111676 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:06.111692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.112645 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.112649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:06.113807 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:06.113821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.114528 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.114533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:06.115110 (+ 577us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.115728 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:06.115747 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:06.117788 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:06.120959 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:06.120967 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:06.120982 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:06.121482 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:06.121887 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:06.121892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:06.121895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:06.122116 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:06.122701 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:06.122708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:06.122979 (+ 271us) client_negotiation.cc:769] Sending connection context
0925 10:32:06.123195 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 10:32:06.123554 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":317,"threads_started":1}
I20260925 10:32:06.126015 1419 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35835
I20260925 10:32:06.126348 1419 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:06.127007 1419 heartbeater.cc:511] Master 127.0.97.62:35835 requested a full tablet report, sending...
I20260925 10:32:06.128275 963 ts_manager.cc:194] Registered new tserver with Master: 9d2d211d2e18495aa7098b3b15011f22 (127.0.97.3:44807)
I20260925 10:32:06.129005 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43833
I20260925 10:32:06.134151 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:32:06.151921 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.139951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.0.97.62:35835)
0925 10:32:06.140170 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:06.140178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:06.140468 (+ 290us) server_negotiation.cc:407] Connection header received
0925 10:32:06.140589 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:06.140592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:06.140648 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:06.140731 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:06.142972 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.143637 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.144831 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.145224 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.145310 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.147400 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:06.147405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:06.147408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:06.147436 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:06.149754 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:06.150284 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:06.150287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:06.150290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:06.150449 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:06.150754 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:06.150757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:06.150759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:06.151039 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:06.151204 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:06.151462 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 10:32:06.151603 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:32:06.157819 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:32:06.167200 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:32:06.187615 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.178372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44807 (local address 127.0.0.1:48802)
0925 10:32:06.179017 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:06.179064 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:06.179235 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:06.180341 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:06.180351 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:06.180383 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:06.180780 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:06.180795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.182837 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.182843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:06.185121 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:06.185133 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.186921 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.186928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:06.187092 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.187132 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 10:32:06.187211 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:32:06.187339 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":24,"thread_start_us":412,"threads_started":1}
I20260925 10:32:06.188807 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.178569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.0.97.3:44807)
0925 10:32:06.179742 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:06.179751 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:06.179783 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:32:06.179876 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:06.179886 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:06.180094 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:06.180331 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:06.181016 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.182692 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.185290 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.186704 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.188058 (+ 1354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.188224 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:06.188279 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 10:32:06.188451 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1011,"thread_start_us":867,"threads_started":1}
I20260925 10:32:06.189361 1152 heartbeater.cc:503] Master 127.0.97.62:35835 was elected leader, sending a full tablet report...
I20260925 10:32:06.190061 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.179507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38525 (local address 127.0.0.1:47606)
0925 10:32:06.179976 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:06.179991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:06.180108 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:06.181476 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:06.181484 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:06.181508 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:06.181888 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:06.181900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.184355 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.184360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:06.186884 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:06.186898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.189222 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.189227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:06.189384 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.189419 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:32:06.189507 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:32:06.189621 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":2,"thread_start_us":629,"threads_started":1}
I20260925 10:32:06.190780 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.179958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.0.97.2:38525)
0925 10:32:06.180893 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:06.180904 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:06.180931 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:32:06.181023 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:06.181030 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:06.181270 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:06.182536 (+ 1266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:06.182767 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.184217 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.187027 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.188798 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.188915 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.189079 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:06.190261 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0925 10:32:06.190469 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":952,"threads_started":1}
I20260925 10:32:06.191535 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.180456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44663 (local address 127.0.0.1:49990)
0925 10:32:06.181499 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:06.181520 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:06.181658 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:06.182932 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:06.182936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:06.182964 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:06.183274 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:06.183282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.184702 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.184707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:06.186674 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:06.186693 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.188824 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.188833 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:06.189228 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.189261 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 10:32:06.189344 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 10:32:06.189524 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":763,"threads_started":1}
I20260925 10:32:06.192278 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.180895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49990 (local address 127.0.97.1:44663)
0925 10:32:06.182399 (+ 1504us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:06.182406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:06.182432 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:32:06.182515 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:06.182522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:06.182733 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:06.183410 (+ 677us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:06.183556 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.184561 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.186841 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.188253 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.188765 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.188987 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:06.191869 (+ 2882us) server_negotiation.cc:299] Negotiation successful
0925 10:32:06.192010 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1291,"thread_start_us":1159,"threads_started":1}
I20260925 10:32:06.192723 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 53f53ea66f8d474e992681c6496ac281 (DEFAULT_TABLE table=test-table [id=f2cafca630b84f898337225749c32cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:06.193886 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53f53ea66f8d474e992681c6496ac281. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:06.193980 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 53f53ea66f8d474e992681c6496ac281 (DEFAULT_TABLE table=test-table [id=f2cafca630b84f898337225749c32cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:06.195408 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53f53ea66f8d474e992681c6496ac281. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:06.195634 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 53f53ea66f8d474e992681c6496ac281 (DEFAULT_TABLE table=test-table [id=f2cafca630b84f898337225749c32cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:06.196892 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53f53ea66f8d474e992681c6496ac281. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:06.204694 1437 tablet_bootstrap.cc:492] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22: Bootstrap starting.
I20260925 10:32:06.208134 1437 tablet_bootstrap.cc:654] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:06.209689 1437 log.cc:824] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:06.209789 1438 tablet_bootstrap.cc:492] T 53f53ea66f8d474e992681c6496ac281 P d91bd616b36b46c4986c32c107720d9b: Bootstrap starting.
I20260925 10:32:06.211102 1440 tablet_bootstrap.cc:492] T 53f53ea66f8d474e992681c6496ac281 P 27e79e60a954482ea9b895036fed3040: Bootstrap starting.
I20260925 10:32:06.213402 1438 tablet_bootstrap.cc:654] T 53f53ea66f8d474e992681c6496ac281 P d91bd616b36b46c4986c32c107720d9b: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:06.214164 1437 tablet_bootstrap.cc:492] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22: No bootstrap required, opened a new log
I20260925 10:32:06.214452 1437 ts_tablet_manager.cc:1397] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260925 10:32:06.214593 1438 log.cc:824] T 53f53ea66f8d474e992681c6496ac281 P d91bd616b36b46c4986c32c107720d9b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:06.214664 1440 tablet_bootstrap.cc:654] T 53f53ea66f8d474e992681c6496ac281 P 27e79e60a954482ea9b895036fed3040: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:06.215823 1440 log.cc:824] T 53f53ea66f8d474e992681c6496ac281 P 27e79e60a954482ea9b895036fed3040: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:06.217586 1438 tablet_bootstrap.cc:492] T 53f53ea66f8d474e992681c6496ac281 P d91bd616b36b46c4986c32c107720d9b: No bootstrap required, opened a new log
I20260925 10:32:06.217826 1438 ts_tablet_manager.cc:1397] T 53f53ea66f8d474e992681c6496ac281 P d91bd616b36b46c4986c32c107720d9b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 10:32:06.218528 1440 tablet_bootstrap.cc:492] T 53f53ea66f8d474e992681c6496ac281 P 27e79e60a954482ea9b895036fed3040: No bootstrap required, opened a new log
I20260925 10:32:06.218758 1440 ts_tablet_manager.cc:1397] T 53f53ea66f8d474e992681c6496ac281 P 27e79e60a954482ea9b895036fed3040: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 10:32:06.224586 1437 raft_consensus.cc:348] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2d211d2e18495aa7098b3b15011f22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44807 } } peers { permanent_uuid: "d91bd616b36b46c4986c32c107720d9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38525 } } peers { permanent_uuid: "27e79e60a954482ea9b895036fed3040" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44663 } }
I20260925 10:32:06.224973 1437 raft_consensus.cc:374] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:06.225060 1437 raft_consensus.cc:729] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2d211d2e18495aa7098b3b15011f22, State: Initialized, Role: FOLLOWER
I20260925 10:32:06.225680 1437 consensus_queue.cc:260] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2d211d2e18495aa7098b3b15011f22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44807 } } peers { permanent_uuid: "d91bd616b36b46c4986c32c107720d9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38525 } } peers { permanent_uuid: "27e79e60a954482ea9b895036fed3040" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44663 } }
I20260925 10:32:06.225683 1440 raft_consensus.cc:348] T 53f53ea66f8d474e992681c6496ac281 P 27e79e60a954482ea9b895036fed3040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2d211d2e18495aa7098b3b15011f22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44807 } } peers { permanent_uuid: "d91bd616b36b46c4986c32c107720d9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38525 } } peers { permanent_uuid: "27e79e60a954482ea9b895036fed3040" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44663 } }
I20260925 10:32:06.226426 1440 raft_consensus.cc:374] T 53f53ea66f8d474e992681c6496ac281 P 27e79e60a954482ea9b895036fed3040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:06.226545 1440 raft_consensus.cc:729] T 53f53ea66f8d474e992681c6496ac281 P 27e79e60a954482ea9b895036fed3040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e79e60a954482ea9b895036fed3040, State: Initialized, Role: FOLLOWER
I20260925 10:32:06.227084 1440 consensus_queue.cc:260] T 53f53ea66f8d474e992681c6496ac281 P 27e79e60a954482ea9b895036fed3040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2d211d2e18495aa7098b3b15011f22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44807 } } peers { permanent_uuid: "d91bd616b36b46c4986c32c107720d9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38525 } } peers { permanent_uuid: "27e79e60a954482ea9b895036fed3040" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44663 } }
I20260925 10:32:06.228612 1438 raft_consensus.cc:348] T 53f53ea66f8d474e992681c6496ac281 P d91bd616b36b46c4986c32c107720d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2d211d2e18495aa7098b3b15011f22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44807 } } peers { permanent_uuid: "d91bd616b36b46c4986c32c107720d9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38525 } } peers { permanent_uuid: "27e79e60a954482ea9b895036fed3040" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44663 } }
I20260925 10:32:06.229077 1438 raft_consensus.cc:374] T 53f53ea66f8d474e992681c6496ac281 P d91bd616b36b46c4986c32c107720d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:06.229187 1438 raft_consensus.cc:729] T 53f53ea66f8d474e992681c6496ac281 P d91bd616b36b46c4986c32c107720d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d91bd616b36b46c4986c32c107720d9b, State: Initialized, Role: FOLLOWER
I20260925 10:32:06.229763 1438 consensus_queue.cc:260] T 53f53ea66f8d474e992681c6496ac281 P d91bd616b36b46c4986c32c107720d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2d211d2e18495aa7098b3b15011f22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44807 } } peers { permanent_uuid: "d91bd616b36b46c4986c32c107720d9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38525 } } peers { permanent_uuid: "27e79e60a954482ea9b895036fed3040" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44663 } }
I20260925 10:32:06.231035 1286 heartbeater.cc:503] Master 127.0.97.62:35835 was elected leader, sending a full tablet report...
I20260925 10:32:06.231786 1438 ts_tablet_manager.cc:1428] T 53f53ea66f8d474e992681c6496ac281 P d91bd616b36b46c4986c32c107720d9b: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260925 10:32:06.232683 1419 heartbeater.cc:503] Master 127.0.97.62:35835 was elected leader, sending a full tablet report...
I20260925 10:32:06.233292 1437 ts_tablet_manager.cc:1428] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22: Time spent starting tablet: real 0.019s user 0.015s sys 0.003s
I20260925 10:32:06.234936 1440 ts_tablet_manager.cc:1428] T 53f53ea66f8d474e992681c6496ac281 P 27e79e60a954482ea9b895036fed3040: Time spent starting tablet: real 0.016s user 0.007s sys 0.009s
I20260925 10:32:06.247419 1443 raft_consensus.cc:482] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:32:06.247648 1443 raft_consensus.cc:504] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2d211d2e18495aa7098b3b15011f22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44807 } } peers { permanent_uuid: "d91bd616b36b46c4986c32c107720d9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38525 } } peers { permanent_uuid: "27e79e60a954482ea9b895036fed3040" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44663 } }
I20260925 10:32:06.249033 1443 leader_election.cc:288] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d91bd616b36b46c4986c32c107720d9b (127.0.97.2:38525), 27e79e60a954482ea9b895036fed3040 (127.0.97.1:44663)
I20260925 10:32:06.254465 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.249365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38525 (local address 127.0.97.3:39635)
0925 10:32:06.249579 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:06.249598 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:06.249719 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:06.250028 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:06.250032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:06.250056 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:06.250286 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:06.250319 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.251311 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.251318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:06.252597 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:06.252608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.253727 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.253733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:06.253916 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.253938 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 10:32:06.254046 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:32:06.254171 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 10:32:06.254477 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.249333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39635 (local address 127.0.97.2:38525)
0925 10:32:06.249525 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:06.249529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:06.249664 (+ 135us) server_negotiation.cc:407] Connection header received
0925 10:32:06.249848 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:06.249854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:06.249914 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:06.250013 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:06.250460 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.251156 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.252770 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.253568 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.253713 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.253803 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:06.254067 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 10:32:06.254186 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:32:06.255342 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f53ea66f8d474e992681c6496ac281" candidate_uuid: "9d2d211d2e18495aa7098b3b15011f22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d91bd616b36b46c4986c32c107720d9b" is_pre_election: true
I20260925 10:32:06.255872 1241 raft_consensus.cc:2454] T 53f53ea66f8d474e992681c6496ac281 P d91bd616b36b46c4986c32c107720d9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d2d211d2e18495aa7098b3b15011f22 in term 0.
I20260925 10:32:06.256028 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.250031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38787 (local address 127.0.97.1:44663)
0925 10:32:06.250261 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:06.250269 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:06.251083 (+ 814us) server_negotiation.cc:407] Connection header received
0925 10:32:06.251257 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:06.251261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:06.251308 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:06.251398 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:06.251946 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.252614 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.254168 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.255147 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.255293 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.255384 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:06.255709 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 10:32:06.255840 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:32:06.256660 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.249801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44663 (local address 127.0.97.3:38787)
0925 10:32:06.250578 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:06.251005 (+ 427us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:06.251134 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:06.251412 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:06.251416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:06.251437 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:06.251804 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:06.251812 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.252765 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.252770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:06.254022 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:06.254032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.255326 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.255332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:06.255510 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.255538 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 10:32:06.256186 (+ 648us) client_negotiation.cc:240] Negotiation successful
0925 10:32:06.256304 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":500,"threads_started":1}
I20260925 10:32:06.256600 1308 leader_election.cc:302] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d2d211d2e18495aa7098b3b15011f22, d91bd616b36b46c4986c32c107720d9b; no voters:
I20260925 10:32:06.257026 1443 raft_consensus.cc:2781] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:32:06.257165 1443 raft_consensus.cc:482] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:32:06.257275 1443 raft_consensus.cc:3037] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:06.257730 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f53ea66f8d474e992681c6496ac281" candidate_uuid: "9d2d211d2e18495aa7098b3b15011f22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27e79e60a954482ea9b895036fed3040" is_pre_election: true
I20260925 10:32:06.258152 1107 raft_consensus.cc:2454] T 53f53ea66f8d474e992681c6496ac281 P 27e79e60a954482ea9b895036fed3040 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d2d211d2e18495aa7098b3b15011f22 in term 0.
I20260925 10:32:06.259995 1443 raft_consensus.cc:504] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2d211d2e18495aa7098b3b15011f22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44807 } } peers { permanent_uuid: "d91bd616b36b46c4986c32c107720d9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38525 } } peers { permanent_uuid: "27e79e60a954482ea9b895036fed3040" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44663 } }
I20260925 10:32:06.260494 1443 leader_election.cc:288] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [CANDIDATE]: Term 1 election: Requested vote from peers d91bd616b36b46c4986c32c107720d9b (127.0.97.2:38525), 27e79e60a954482ea9b895036fed3040 (127.0.97.1:44663)
I20260925 10:32:06.260911 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f53ea66f8d474e992681c6496ac281" candidate_uuid: "9d2d211d2e18495aa7098b3b15011f22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d91bd616b36b46c4986c32c107720d9b"
I20260925 10:32:06.261018 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f53ea66f8d474e992681c6496ac281" candidate_uuid: "9d2d211d2e18495aa7098b3b15011f22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27e79e60a954482ea9b895036fed3040"
I20260925 10:32:06.261102 1241 raft_consensus.cc:3037] T 53f53ea66f8d474e992681c6496ac281 P d91bd616b36b46c4986c32c107720d9b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:06.261230 1107 raft_consensus.cc:3037] T 53f53ea66f8d474e992681c6496ac281 P 27e79e60a954482ea9b895036fed3040 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:06.264045 1107 raft_consensus.cc:2454] T 53f53ea66f8d474e992681c6496ac281 P 27e79e60a954482ea9b895036fed3040 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d2d211d2e18495aa7098b3b15011f22 in term 1.
I20260925 10:32:06.264092 1241 raft_consensus.cc:2454] T 53f53ea66f8d474e992681c6496ac281 P d91bd616b36b46c4986c32c107720d9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d2d211d2e18495aa7098b3b15011f22 in term 1.
I20260925 10:32:06.264374 1308 leader_election.cc:302] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27e79e60a954482ea9b895036fed3040, 9d2d211d2e18495aa7098b3b15011f22; no voters:
I20260925 10:32:06.264652 1443 raft_consensus.cc:2781] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:06.265076 1443 raft_consensus.cc:686] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [term 1 LEADER]: Becoming Leader. State: Replica: 9d2d211d2e18495aa7098b3b15011f22, State: Running, Role: LEADER
I20260925 10:32:06.265594 1443 consensus_queue.cc:237] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2d211d2e18495aa7098b3b15011f22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44807 } } peers { permanent_uuid: "d91bd616b36b46c4986c32c107720d9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38525 } } peers { permanent_uuid: "27e79e60a954482ea9b895036fed3040" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44663 } }
I20260925 10:32:06.270192 963 catalog_manager.cc:5495] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d2d211d2e18495aa7098b3b15011f22 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9d2d211d2e18495aa7098b3b15011f22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2d211d2e18495aa7098b3b15011f22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d91bd616b36b46c4986c32c107720d9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27e79e60a954482ea9b895036fed3040" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44663 } health_report { overall_health: UNKNOWN } } }
Sep 25 10:32:06 dist-test-slave-dxdd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:32:06.291175 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.277595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.0.97.1:44663)
0925 10:32:06.277797 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:06.277805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:06.278262 (+ 457us) server_negotiation.cc:407] Connection header received
0925 10:32:06.278452 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:06.278456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:06.278538 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:06.278614 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:06.279957 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.280683 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.281885 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.282431 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.282518 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.286427 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:06.286439 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:06.286452 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:06.286480 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:06.288954 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:06.289509 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:06.289515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:06.289517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:06.289618 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:06.289909 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:06.289912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:06.289914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:06.290201 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:06.290475 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:06.290674 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 10:32:06.290816 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 10:32:06 dist-test-slave-dxdd krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:32:06 dist-test-slave-dxdd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:32:06.333447 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.320168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50018 (local address 127.0.97.1:44663)
0925 10:32:06.320409 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:06.320419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:06.320542 (+ 123us) server_negotiation.cc:407] Connection header received
0925 10:32:06.320701 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:06.320704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:06.320764 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:06.320855 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:06.322085 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.322914 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.324135 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.324596 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.324716 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.328799 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:06.328804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:06.328807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:06.328839 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:06.331210 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:06.331848 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:06.331852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:06.331855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:06.331966 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:06.332300 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:06.332303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:06.332305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:06.332530 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:06.332761 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:06.333062 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 10:32:06.333214 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260925 10:32:06.355585 1420 tablet.cc:2369] T 53f53ea66f8d474e992681c6496ac281 P 9d2d211d2e18495aa7098b3b15011f22: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 10:32:06 dist-test-slave-dxdd krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:32:06 dist-test-slave-dxdd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
W20260925 10:32:06.372958 1287 tablet.cc:2369] T 53f53ea66f8d474e992681c6496ac281 P d91bd616b36b46c4986c32c107720d9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:32:06.375933 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.362621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.0.97.1:44663)
0925 10:32:06.362857 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:06.362865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:06.363181 (+ 316us) server_negotiation.cc:407] Connection header received
0925 10:32:06.363376 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:06.363381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:06.363453 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:06.363573 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:06.364860 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.365491 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.366839 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.367315 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.367405 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.370917 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:06.370922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:06.370925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:06.370953 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:06.373266 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:06.373953 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:06.373957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:06.373959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:06.374063 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:06.374404 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:06.374407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:06.374409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:06.375050 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:06.375236 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:06.375424 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 10:32:06.375580 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 10:32:06.377529 1067 tablet_service.cc:2908] Rejecting scan request for tablet 53f53ea66f8d474e992681c6496ac281: Uninitialized: clean time has not yet been initialized
I20260925 10:32:06.380465 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 1024
I20260925 10:32:06.392438 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.381300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.0.97.1:44663)
0925 10:32:06.381462 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:06.381468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:06.381758 (+ 290us) server_negotiation.cc:407] Connection header received
0925 10:32:06.381951 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:06.381955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:06.382029 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:06.382131 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:06.383254 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.383892 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.385108 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.385515 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.385674 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.387763 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:06.387769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:06.387776 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:06.387809 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:06.389950 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:06.390604 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:06.390611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:06.390614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:06.390731 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:06.391096 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:06.391101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:06.391104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:06.391376 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:06.391676 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:06.391927 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 10:32:06.392144 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 10:32:06.392989 1147 generic_service.cc:196] Checking for leaks (request via RPC)
W20260925 10:32:06.477326 1153 tablet.cc:2369] T 53f53ea66f8d474e992681c6496ac281 P 27e79e60a954482ea9b895036fed3040: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:32:06.477942 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 1024
I20260925 10:32:06.494376 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 1158
Sep 25 10:32:06 dist-test-slave-dxdd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:32:06.508208 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.495145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47610 (local address 127.0.97.2:38525)
0925 10:32:06.495336 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:06.495343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:06.495363 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:32:06.495425 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:06.495428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:06.495500 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:06.495610 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:06.496778 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.497507 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.498825 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.499217 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.499341 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.503107 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:06.503118 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:06.503133 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:06.503162 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:06.505588 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:06.506240 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:06.506254 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:06.506257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:06.506425 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:06.506799 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:06.506805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:06.506809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:06.507149 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:06.507485 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:06.507691 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 10:32:06.507903 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:32:06.508845 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:06.597668 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 1158
I20260925 10:32:06.613222 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 1291
Sep 25 10:32:06 dist-test-slave-dxdd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:32:06.626607 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.613972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48816 (local address 127.0.97.3:44807)
0925 10:32:06.614144 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:06.614149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:06.614166 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:32:06.614300 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:06.614335 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:06.614402 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:06.614500 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:06.615645 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.616290 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.617427 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:06.617807 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:06.617895 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:06.621651 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:06.621663 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:06.621678 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:06.621708 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:06.624223 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:06.624776 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:06.624782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:06.624784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:06.624910 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:06.625198 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:06.625202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:06.625206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:06.625569 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:06.625812 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:06.626016 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 10:32:06.626179 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:32:06.627202 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:06.712770 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 1291
I20260925 10:32:06.712803 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:06.712382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44663 (local address 127.0.97.3:47105)
0925 10:32:06.712542 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:06.712662 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 10:32:06.728348 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 931
I20260925 10:32:06.729416 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:06.832912 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 931
2026-09-25T10:32:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (7341 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:32:06 dist-test-slave-dxdd krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:32:06 dist-test-slave-dxdd krb5kdc[1466](info): set up 2 sockets
Sep 25 10:32:06 dist-test-slave-dxdd krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260925 10:32:07.898126 388 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:32:08 dist-test-slave-dxdd krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:32:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:32:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:32:08.069149 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35543
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:08.371330 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:08.371618 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:08.371706 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:08.378649 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:08.378747 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:32:08.378788 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:08.378840 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:08.378890 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:08.378949 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:32:08.388423 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35543
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:08.389604 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:08.390960 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:08.402284 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:08.402352 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:08.402391 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:08.403026 1479 server_base.cc:1034] running on GCE node
I20260925 10:32:08.403787 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:08.405079 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:08.406358 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790332328406249 us; error 49 us; skew 500 ppm
I20260925 10:32:08.406689 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:08 dist-test-slave-dxdd krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:08.409597 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:32:08.411079 1479 webserver.cc:460] Webserver started at http://127.0.97.62:32971/ using document root <none> and password file <none>
I20260925 10:32:08.411739 1479 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:08.411870 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:08.412187 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:08.415032 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "914c2911f1014c30bac107166544e0ee"
format_stamp: "Formatted at 2026-09-25 10:32:08 on dist-test-slave-dxdd"
server_key: "337e8fde8dec6cf79581039199b2f90c"
server_key_iv: "b560fe154f2913331c5410ff06f181b9"
server_key_version: "encryptionkey@0"
I20260925 10:32:08.415813 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "914c2911f1014c30bac107166544e0ee"
format_stamp: "Formatted at 2026-09-25 10:32:08 on dist-test-slave-dxdd"
server_key: "337e8fde8dec6cf79581039199b2f90c"
server_key_iv: "b560fe154f2913331c5410ff06f181b9"
server_key_version: "encryptionkey@0"
I20260925 10:32:08.420463 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:32:08.423604 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:08.424783 1479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 10:32:08.424952 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "914c2911f1014c30bac107166544e0ee"
format_stamp: "Formatted at 2026-09-25 10:32:08 on dist-test-slave-dxdd"
server_key: "337e8fde8dec6cf79581039199b2f90c"
server_key_iv: "b560fe154f2913331c5410ff06f181b9"
server_key_version: "encryptionkey@0"
I20260925 10:32:08.425160 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:08.445413 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:08.449146 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:08.449431 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:08.470052 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35543
I20260925 10:32:08.470077 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35543 every 8 connection(s)
I20260925 10:32:08.471447 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:32:08.474632 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 1479
I20260925 10:32:08.474859 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:32:08.475404 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:08.475567 388 external_mini_cluster.cc:1428] Setting key 1954a5f4a7c646ddbfab29bbb398d326
I20260925 10:32:08.489436 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee: Bootstrap starting.
Sep 25 10:32:08 dist-test-slave-dxdd krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:08.493155 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:08.494464 1556 log.cc:824] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:08.497668 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee: No bootstrap required, opened a new log
I20260925 10:32:08.499153 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:08.477566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.0.97.62:35543)
0925 10:32:08.478396 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:08.478411 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:08.478463 (+ 52us) server_negotiation.cc:407] Connection header received
0925 10:32:08.480561 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:08.480589 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:08.481094 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:08.481541 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:08.482795 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:08.484701 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:08.485902 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:08.486971 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:08.487157 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:08.490947 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:08.490959 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:08.490984 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:08.491032 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:08.494727 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:08.495540 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:08.495546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:08.495551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:08.495741 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:08.496113 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:08.496118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:08.496121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:08.496751 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:08.497036 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:08.497415 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 10:32:08.497817 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":334,"threads_started":1}
I20260925 10:32:08.506187 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914c2911f1014c30bac107166544e0ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35543 } }
I20260925 10:32:08.506582 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:08.506660 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914c2911f1014c30bac107166544e0ee, State: Initialized, Role: FOLLOWER
I20260925 10:32:08.507095 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914c2911f1014c30bac107166544e0ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35543 } }
I20260925 10:32:08.507344 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:32:08.507493 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:32:08.507671 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:08.509696 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914c2911f1014c30bac107166544e0ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35543 } }
I20260925 10:32:08.510092 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 914c2911f1014c30bac107166544e0ee; no voters:
I20260925 10:32:08.510619 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:32:08.510710 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:08.511396 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [term 1 LEADER]: Becoming Leader. State: Replica: 914c2911f1014c30bac107166544e0ee, State: Running, Role: LEADER
I20260925 10:32:08.511888 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914c2911f1014c30bac107166544e0ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35543 } }
I20260925 10:32:08.512512 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:08.514428 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 914c2911f1014c30bac107166544e0ee. Latest consensus state: current_term: 1 leader_uuid: "914c2911f1014c30bac107166544e0ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914c2911f1014c30bac107166544e0ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35543 } } }
I20260925 10:32:08.514834 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:08.515342 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "914c2911f1014c30bac107166544e0ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914c2911f1014c30bac107166544e0ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35543 } } }
I20260925 10:32:08.515614 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:08.515679 1569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:08.521150 1569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:08.528894 1569 catalog_manager.cc:1347] Generated new cluster ID: c7a5f18e0ce5491985394c2595b847fb
I20260925 10:32:08.529031 1569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:08.553706 1569 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:32:08.554983 1569 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:08.561265 1569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee: Generated new TSK 0
I20260925 10:32:08.562184 1569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:32:08.611954 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35543
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:08.919272 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:08.919581 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:08.919752 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:08.926856 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:08.927013 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:08.927197 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:32:08.927345 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:32:08.936923 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:08.938206 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:08.939579 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:08.952236 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:08.952334 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:08.952620 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:08.953014 1582 server_base.cc:1034] running on GCE node
I20260925 10:32:08.953576 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:08.954479 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:08.955716 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790332328955661 us; error 33 us; skew 500 ppm
I20260925 10:32:08.956041 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:08 dist-test-slave-dxdd krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:08.964643 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:32:08.966066 1582 webserver.cc:460] Webserver started at http://127.0.97.1:34711/ using document root <none> and password file <none>
I20260925 10:32:08.966847 1582 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:08.966988 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:08.967319 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:08.970177 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "a3f05340707d4837a46fa30eca0a59e5"
format_stamp: "Formatted at 2026-09-25 10:32:08 on dist-test-slave-dxdd"
server_key: "3b91196e1f2479700edfc72ad7358623"
server_key_iv: "6cbbaf6825fa6e6ace2d0eb12be2b7e0"
server_key_version: "encryptionkey@0"
I20260925 10:32:08.971017 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a3f05340707d4837a46fa30eca0a59e5"
format_stamp: "Formatted at 2026-09-25 10:32:08 on dist-test-slave-dxdd"
server_key: "3b91196e1f2479700edfc72ad7358623"
server_key_iv: "6cbbaf6825fa6e6ace2d0eb12be2b7e0"
server_key_version: "encryptionkey@0"
I20260925 10:32:08.976109 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:32:08.979504 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:08.980706 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:32:08.980870 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "a3f05340707d4837a46fa30eca0a59e5"
format_stamp: "Formatted at 2026-09-25 10:32:08 on dist-test-slave-dxdd"
server_key: "3b91196e1f2479700edfc72ad7358623"
server_key_iv: "6cbbaf6825fa6e6ace2d0eb12be2b7e0"
server_key_version: "encryptionkey@0"
I20260925 10:32:08.981105 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:09.005196 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:09.009338 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:09.009611 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:09.012259 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:09.012362 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:09.012586 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:09.012671 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:32:09 dist-test-slave-dxdd krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:09.036561 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:09.012698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52215 (local address 127.0.97.62:35543)
0925 10:32:09.013160 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:09.013167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:09.014515 (+ 1348us) server_negotiation.cc:407] Connection header received
0925 10:32:09.017467 (+ 2952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:09.017476 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:09.017560 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:09.017672 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:09.020171 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:09.020856 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:09.023780 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:09.024153 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:09.024257 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:09.029185 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:09.029190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:09.029217 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:09.029262 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:09.032705 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:09.033690 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:09.033698 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:09.033704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:09.033846 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:09.034453 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:09.034462 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:09.034468 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:09.034916 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:09.035099 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:09.035730 (+ 631us) server_negotiation.cc:299] Negotiation successful
0925 10:32:09.036010 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":238,"threads_started":1}
I20260925 10:32:09.036873 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:09.013319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35543 (local address 127.0.97.1:52215)
0925 10:32:09.014350 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:09.014401 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:09.017131 (+ 2730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:09.017984 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:09.017995 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:09.018929 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:09.019963 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:09.020002 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:09.021004 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:09.021012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:09.023629 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:09.023648 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:09.024396 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:09.024405 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:09.024730 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:09.025433 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:09.025469 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:09.028906 (+ 3437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:09.032837 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:09.032849 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:09.032864 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:09.033496 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:09.033967 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:09.033971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:09.033975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:09.034293 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:09.035046 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:09.035052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:09.035351 (+ 299us) client_negotiation.cc:769] Sending connection context
0925 10:32:09.035561 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 10:32:09.035905 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":377,"threads_started":1}
I20260925 10:32:09.063097 1656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:09.042809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35543 (local address 127.0.97.1:43401)
0925 10:32:09.043403 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:09.043418 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:09.043559 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:09.043870 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:09.043874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:09.044297 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:09.044557 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:09.044572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:09.045779 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:09.045787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:09.047971 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:09.047987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:09.049491 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:09.049499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:09.049766 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:09.054677 (+ 4911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:09.054704 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:09.055978 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:09.059972 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:09.059978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:09.059983 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:09.060642 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:09.061044 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:09.061049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:09.061052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:09.061253 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:09.061914 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:09.061918 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:09.062096 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 10:32:09.062238 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 10:32:09.062593 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":341,"threads_started":1}
I20260925 10:32:09.063620 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:09.042613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43401 (local address 127.0.97.62:35543)
0925 10:32:09.042781 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:09.042788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:09.043497 (+ 709us) server_negotiation.cc:407] Connection header received
0925 10:32:09.043683 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:09.043688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:09.043756 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:09.043850 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:09.044704 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:09.045634 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:09.048166 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:09.049046 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:09.049154 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:09.056175 (+ 7021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:09.056181 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:09.056186 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:09.056220 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:09.059844 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:09.060778 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:09.060784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:09.060786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:09.060950 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:09.061382 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:09.061387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:09.061390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:09.061796 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:09.061950 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:09.063052 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0925 10:32:09.063275 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:32:09.079550 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39545
I20260925 10:32:09.079567 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39545 every 8 connection(s)
I20260925 10:32:09.080694 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:32:09.082736 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35543
I20260925 10:32:09.082916 1718 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:09.083454 1718 heartbeater.cc:511] Master 127.0.97.62:35543 requested a full tablet report, sending...
I20260925 10:32:09.085247 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 1582
I20260925 10:32:09.085232 1511 ts_manager.cc:194] Registered new tserver with Master: a3f05340707d4837a46fa30eca0a59e5 (127.0.97.1:39545)
I20260925 10:32:09.085461 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:32:09.086082 388 external_mini_cluster.cc:1428] Setting key 11bb3344350e535a24f5ed00fd1fac09
I20260925 10:32:09.087038 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52215
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:32:09.123049 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35543
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:09.427587 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:09.427826 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:09.427922 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:09.434901 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:09.434993 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:09.435075 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:32:09.435149 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:32:09.444509 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:09.445631 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:09.446995 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:09.459316 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:09.459372 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:09.459312 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:09.460127 1722 server_base.cc:1034] running on GCE node
I20260925 10:32:09.460589 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:09.461328 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:09.462551 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790332329462493 us; error 38 us; skew 500 ppm
I20260925 10:32:09.462893 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:09 dist-test-slave-dxdd krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:09.465786 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:32:09.467097 1722 webserver.cc:460] Webserver started at http://127.0.97.2:44765/ using document root <none> and password file <none>
I20260925 10:32:09.467689 1722 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:09.467768 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:09.468016 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:09.470696 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "18ed0d768ca64db88c485109e83ee0ac"
format_stamp: "Formatted at 2026-09-25 10:32:09 on dist-test-slave-dxdd"
server_key: "cb8ab5f1533ef3684833842ad830aa24"
server_key_iv: "ca7f0a86a8f0f0ef17a5db0cbe3c0ded"
server_key_version: "encryptionkey@0"
I20260925 10:32:09.471436 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "18ed0d768ca64db88c485109e83ee0ac"
format_stamp: "Formatted at 2026-09-25 10:32:09 on dist-test-slave-dxdd"
server_key: "cb8ab5f1533ef3684833842ad830aa24"
server_key_iv: "ca7f0a86a8f0f0ef17a5db0cbe3c0ded"
server_key_version: "encryptionkey@0"
I20260925 10:32:09.476366 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:32:09.479700 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:09.481007 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:32:09.481161 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "18ed0d768ca64db88c485109e83ee0ac"
format_stamp: "Formatted at 2026-09-25 10:32:09 on dist-test-slave-dxdd"
server_key: "cb8ab5f1533ef3684833842ad830aa24"
server_key_iv: "ca7f0a86a8f0f0ef17a5db0cbe3c0ded"
server_key_version: "encryptionkey@0"
I20260925 10:32:09.481390 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:09.495801 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:09.499329 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:09.499572 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:09.501833 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:09.501945 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:09.502064 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:09.502162 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:32:09 dist-test-slave-dxdd krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:09.525846 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:09.503111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35543 (local address 127.0.97.2:41959)
0925 10:32:09.503855 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:09.503887 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:09.505657 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:09.506528 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:09.506548 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:09.507339 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:09.508243 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:09.508261 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:09.509620 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:09.509624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:09.511852 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:09.511873 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:09.512873 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:09.512882 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:09.513195 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:09.514002 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:09.514025 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:09.517460 (+ 3435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:09.520917 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:09.520925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:09.520943 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:09.521478 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:09.521923 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:09.521930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:09.521940 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:09.522230 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:09.522901 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:09.522911 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:09.523390 (+ 479us) client_negotiation.cc:769] Sending connection context
0925 10:32:09.523703 (+ 313us) client_negotiation.cc:240] Negotiation successful
0925 10:32:09.524274 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":357,"threads_started":1}
I20260925 10:32:09.526840 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:09.502459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41959 (local address 127.0.97.62:35543)
0925 10:32:09.502621 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:09.502625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:09.504002 (+ 1377us) server_negotiation.cc:407] Connection header received
0925 10:32:09.506010 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:09.506019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:09.506111 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:09.506225 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:09.508421 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:09.509413 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:09.512051 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:09.512633 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:09.512779 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:09.517693 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:09.517698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:09.517701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:09.517731 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:09.520794 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:09.521631 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:09.521636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:09.521641 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:09.521771 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:09.522373 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:09.522378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:09.522382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:09.522779 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:09.522922 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:09.526422 (+ 3500us) server_negotiation.cc:299] Negotiation successful
0925 10:32:09.526576 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 10:32:09.547713 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:09.533128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52281 (local address 127.0.97.62:35543)
0925 10:32:09.533299 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:09.533304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:09.533322 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:32:09.533386 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:09.533391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:09.533458 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:09.533551 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:09.534978 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:09.536481 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:09.537819 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:09.538678 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:09.538826 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:09.540623 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:09.540632 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:09.540640 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:09.540688 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:09.544484 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:09.545119 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:09.545128 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:09.545134 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:09.545282 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:09.545789 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:09.545796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:09.545800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:09.546227 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:09.546399 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:09.547192 (+ 793us) server_negotiation.cc:299] Negotiation successful
0925 10:32:09.547334 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:32:09.548033 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:09.532394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35543 (local address 127.0.97.2:52281)
0925 10:32:09.533139 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:09.533154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:09.533284 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:09.533582 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:09.533587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:09.534287 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:09.534783 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:09.534800 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:09.536638 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:09.536642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:09.537685 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:09.537695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:09.538897 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:09.538903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:09.539086 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:09.539745 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:09.539760 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:09.540417 (+ 657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:09.544609 (+ 4192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:09.544613 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:09.544615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:09.544992 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:09.545469 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:09.545475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:09.545480 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:09.545655 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:09.546385 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:09.546389 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:09.546679 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 10:32:09.546859 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 10:32:09.547529 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":402,"threads_started":1}
I20260925 10:32:09.555799 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35825
I20260925 10:32:09.555817 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35825 every 8 connection(s)
I20260925 10:32:09.556980 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:32:09.558226 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 1722
I20260925 10:32:09.558450 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:32:09.559028 388 external_mini_cluster.cc:1428] Setting key e1a09fdb7914d9426219ae00f21a800e
I20260925 10:32:09.559060 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35543
I20260925 10:32:09.559275 1857 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:09.560030 1857 heartbeater.cc:511] Master 127.0.97.62:35543 requested a full tablet report, sending...
I20260925 10:32:09.561415 1510 ts_manager.cc:194] Registered new tserver with Master: 18ed0d768ca64db88c485109e83ee0ac (127.0.97.2:35825)
I20260925 10:32:09.562264 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41959
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:32:09.597342 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35543
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:09.907148 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:09.907444 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:09.907636 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:09.915118 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:09.915275 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:09.915418 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:32:09.915551 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:32:09.924891 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:09.926040 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:09.927320 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:09.939850 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:09.940007 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:09.940037 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:09.943605 1861 server_base.cc:1034] running on GCE node
I20260925 10:32:09.944159 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:09.944980 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:09.946244 1861 hybrid_clock.cc:648] HybridClock initialized: now 1790332329946182 us; error 41 us; skew 500 ppm
I20260925 10:32:09.946542 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:09 dist-test-slave-dxdd krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:09.949460 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:32:09.950919 1861 webserver.cc:460] Webserver started at http://127.0.97.3:43449/ using document root <none> and password file <none>
I20260925 10:32:09.951570 1861 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:09.951696 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:09.951994 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:09.954941 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "e2c1759857fa4822a2318abd541d8258"
format_stamp: "Formatted at 2026-09-25 10:32:09 on dist-test-slave-dxdd"
server_key: "f45fc0ff2093365d789ad1148dbbb262"
server_key_iv: "34bab6f87c4e7b723162db5cc64a3fcf"
server_key_version: "encryptionkey@0"
I20260925 10:32:09.955698 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e2c1759857fa4822a2318abd541d8258"
format_stamp: "Formatted at 2026-09-25 10:32:09 on dist-test-slave-dxdd"
server_key: "f45fc0ff2093365d789ad1148dbbb262"
server_key_iv: "34bab6f87c4e7b723162db5cc64a3fcf"
server_key_version: "encryptionkey@0"
I20260925 10:32:09.960443 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:32:09.963758 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:09.964984 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 10:32:09.965163 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "e2c1759857fa4822a2318abd541d8258"
format_stamp: "Formatted at 2026-09-25 10:32:09 on dist-test-slave-dxdd"
server_key: "f45fc0ff2093365d789ad1148dbbb262"
server_key_iv: "34bab6f87c4e7b723162db5cc64a3fcf"
server_key_version: "encryptionkey@0"
I20260925 10:32:09.965389 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:09.987144 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:09.991081 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:09.991336 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:09.993804 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:09.993927 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:09.994091 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:09.994179 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:32:10 dist-test-slave-dxdd krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:10.019042 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:09.994167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50939 (local address 127.0.97.62:35543)
0925 10:32:09.994375 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:09.994386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:09.995920 (+ 1534us) server_negotiation.cc:407] Connection header received
0925 10:32:09.998897 (+ 2977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:09.998903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:09.998986 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:09.999065 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:10.001862 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.002840 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.006216 (+ 3376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.007024 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.007144 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.012245 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:10.012249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:10.012253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:10.012283 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:10.015432 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:10.016418 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:10.016422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:10.016424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:10.016670 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:10.017219 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:10.017222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:10.017224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:10.017445 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:10.017540 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:10.018609 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0925 10:32:10.018800 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 10:32:10.021068 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:09.994857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35543 (local address 127.0.97.3:50939)
0925 10:32:09.995734 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:09.995788 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:09.998525 (+ 2737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:09.999723 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:09.999738 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:10.000637 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:10.001645 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.001676 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.003022 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.003030 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.005984 (+ 2954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.006004 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.007489 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.007498 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.007772 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.008619 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:10.008648 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:10.012025 (+ 3377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:10.015566 (+ 3541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:10.015579 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:10.015600 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:10.016296 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:10.016777 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:10.016784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:10.016790 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:10.017109 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:10.017576 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:10.017588 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:10.018119 (+ 531us) client_negotiation.cc:769] Sending connection context
0925 10:32:10.018508 (+ 389us) client_negotiation.cc:240] Negotiation successful
0925 10:32:10.019162 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":374,"threads_started":1}
I20260925 10:32:10.042296 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.030140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48561 (local address 127.0.97.62:35543)
0925 10:32:10.030326 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:10.030334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:10.030461 (+ 127us) server_negotiation.cc:407] Connection header received
0925 10:32:10.030686 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:10.030691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:10.030761 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:10.030859 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:10.031718 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.032450 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.033727 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.034183 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.034302 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.036577 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:10.036583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:10.036587 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:10.036629 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:10.039917 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:10.040502 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:10.040508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:10.040512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:10.040630 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:10.040970 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:10.040975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:10.040978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:10.041226 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:10.041374 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:10.041737 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 10:32:10.041949 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:32:10.042660 1971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.029751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35543 (local address 127.0.97.3:48561)
0925 10:32:10.030364 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:10.030386 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:10.030565 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:10.030877 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:10.030882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:10.031308 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:10.031579 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.031587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.032577 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.032582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.033586 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.033595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.034399 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.034406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.034558 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.035297 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:10.035315 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:10.036421 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:10.040033 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:10.040039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:10.040043 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:10.040394 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:10.040730 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:10.040736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:10.040739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:10.040863 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:10.041346 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:10.041350 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:10.041511 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 10:32:10.041635 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 10:32:10.041942 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":587,"threads_started":1}
I20260925 10:32:10.046850 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34967
I20260925 10:32:10.046888 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34967 every 8 connection(s)
I20260925 10:32:10.048061 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:32:10.049979 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35543
I20260925 10:32:10.050181 1996 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:10.050802 1996 heartbeater.cc:511] Master 127.0.97.62:35543 requested a full tablet report, sending...
I20260925 10:32:10.051966 1510 ts_manager.cc:194] Registered new tserver with Master: e2c1759857fa4822a2318abd541d8258 (127.0.97.3:34967)
I20260925 10:32:10.052876 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50939
I20260925 10:32:10.054544 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 1861
I20260925 10:32:10.054723 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:32:10.055122 388 external_mini_cluster.cc:1428] Setting key de75ead50ab91c7752b0fb3ea7919848
I20260925 10:32:10.057593 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:32:10 dist-test-slave-dxdd krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260925 10:32:10.089775 1718 heartbeater.cc:503] Master 127.0.97.62:35543 was elected leader, sending a full tablet report...
Sep 25 10:32:10 dist-test-slave-dxdd krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:10.102715 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.089652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.0.97.62:35543)
0925 10:32:10.090090 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:10.090096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:10.090117 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:32:10.090291 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:10.090299 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:10.090409 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:10.090516 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:10.091782 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.092482 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.093710 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.094078 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.094227 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.097723 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:10.097728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:10.097731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:10.097760 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:10.100241 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:10.100829 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:10.100836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:10.100840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:10.100965 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:10.101348 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:10.101355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:10.101358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:10.101595 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:10.101828 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:10.102124 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 10:32:10.102393 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 10:32:10.106436 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:32:10.113209 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:32:10.134861 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.125004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35825 (local address 127.0.0.1:55236)
0925 10:32:10.126478 (+ 1474us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:10.126496 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:10.126618 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:10.127481 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:10.127524 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:10.127548 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:10.127992 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.128006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.129842 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.129846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.131982 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.131995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.133938 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.133943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.134141 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.134177 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 10:32:10.134299 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 10:32:10.134485 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1367,"thread_start_us":2290,"threads_started":1}
I20260925 10:32:10.135234 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.124540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39545 (local address 127.0.0.1:36402)
0925 10:32:10.125434 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:10.125486 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:10.125635 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:10.126417 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:10.126424 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:10.126454 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:10.126891 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.126902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.128866 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.128872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.131433 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.131449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.133360 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.133367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.133581 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.134494 (+ 913us) client_negotiation.cc:769] Sending connection context
0925 10:32:10.134553 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 10:32:10.134675 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":311,"threads_started":1}
I20260925 10:32:10.135774 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.125508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.0.97.2:35825)
0925 10:32:10.126170 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:10.126187 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:10.126750 (+ 563us) server_negotiation.cc:407] Connection header received
0925 10:32:10.126852 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:10.126860 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:10.127093 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:10.127278 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:10.128179 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.129696 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.132152 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.133461 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.133607 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.133807 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:10.135357 (+ 1550us) server_negotiation.cc:299] Negotiation successful
0925 10:32:10.135520 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":386,"threads_started":1}
I20260925 10:32:10.136288 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.124736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36402 (local address 127.0.97.1:39545)
0925 10:32:10.125737 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:10.125749 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:10.125781 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:32:10.125893 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:10.125904 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:10.126183 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:10.126419 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:10.127093 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.128723 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.131676 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.133132 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.135606 (+ 2474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.135793 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:10.135954 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 10:32:10.136093 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":713,"threads_started":1}
I20260925 10:32:10.140070 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 9a633b2883464db383e6edc4059553a3 (DEFAULT_TABLE table=test-table [id=55ff213cdd4e40f4a521257fe6eb71ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:10.141288 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a633b2883464db383e6edc4059553a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:10.142035 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 9a633b2883464db383e6edc4059553a3 (DEFAULT_TABLE table=test-table [id=55ff213cdd4e40f4a521257fe6eb71ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:10.143458 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a633b2883464db383e6edc4059553a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:10.146544 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.126560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34967 (local address 127.0.0.1:37496)
0925 10:32:10.135087 (+ 8527us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:10.135103 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:10.135188 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:10.139896 (+ 4708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:10.139901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:10.139926 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:10.140086 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.140096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.142198 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.142206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.144005 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.144018 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.145760 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.145767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.145936 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.145969 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 10:32:10.146082 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 10:32:10.146228 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8429,"thread_start_us":11950,"threads_started":1}
I20260925 10:32:10.147569 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.127063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.0.97.3:34967)
0925 10:32:10.139313 (+ 12250us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:10.139320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:10.139350 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:32:10.139426 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:10.139431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:10.139634 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:10.140237 (+ 603us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:10.140539 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.142033 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.144165 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.145594 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.145722 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.147023 (+ 1301us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:10.147090 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 10:32:10.147262 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12044,"thread_start_us":11891,"threads_started":1}
I20260925 10:32:10.151130 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 9a633b2883464db383e6edc4059553a3 (DEFAULT_TABLE table=test-table [id=55ff213cdd4e40f4a521257fe6eb71ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:10.152452 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a633b2883464db383e6edc4059553a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:10.153395 2015 tablet_bootstrap.cc:492] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac: Bootstrap starting.
I20260925 10:32:10.156518 2015 tablet_bootstrap.cc:654] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:10.157414 2015 log.cc:824] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:10.158339 2016 tablet_bootstrap.cc:492] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5: Bootstrap starting.
I20260925 10:32:10.161810 2015 tablet_bootstrap.cc:492] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac: No bootstrap required, opened a new log
I20260925 10:32:10.162061 2015 ts_tablet_manager.cc:1397] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 10:32:10.162617 2016 tablet_bootstrap.cc:654] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:10.163829 2016 log.cc:824] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:10.165627 2019 tablet_bootstrap.cc:492] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258: Bootstrap starting.
I20260925 10:32:10.166793 2016 tablet_bootstrap.cc:492] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5: No bootstrap required, opened a new log
I20260925 10:32:10.167047 2016 ts_tablet_manager.cc:1397] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260925 10:32:10.169241 2019 tablet_bootstrap.cc:654] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:10.170409 2019 log.cc:824] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:10.172297 2015 raft_consensus.cc:348] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } }
I20260925 10:32:10.172709 2015 raft_consensus.cc:374] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:10.172822 2019 tablet_bootstrap.cc:492] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258: No bootstrap required, opened a new log
I20260925 10:32:10.172851 2015 raft_consensus.cc:729] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ed0d768ca64db88c485109e83ee0ac, State: Initialized, Role: FOLLOWER
I20260925 10:32:10.173097 2019 ts_tablet_manager.cc:1397] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 10:32:10.173492 2015 consensus_queue.cc:260] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } }
I20260925 10:32:10.174959 1857 heartbeater.cc:503] Master 127.0.97.62:35543 was elected leader, sending a full tablet report...
I20260925 10:32:10.176910 2015 ts_tablet_manager.cc:1428] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260925 10:32:10.177640 2016 raft_consensus.cc:348] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } }
I20260925 10:32:10.178113 2016 raft_consensus.cc:374] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:10.178261 2016 raft_consensus.cc:729] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f05340707d4837a46fa30eca0a59e5, State: Initialized, Role: FOLLOWER
I20260925 10:32:10.178951 2016 consensus_queue.cc:260] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } }
I20260925 10:32:10.180496 2016 ts_tablet_manager.cc:1428] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260925 10:32:10.183671 2019 raft_consensus.cc:348] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } }
I20260925 10:32:10.184037 2019 raft_consensus.cc:374] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:10.184131 2019 raft_consensus.cc:729] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c1759857fa4822a2318abd541d8258, State: Initialized, Role: FOLLOWER
I20260925 10:32:10.184688 2019 consensus_queue.cc:260] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } }
I20260925 10:32:10.185690 1996 heartbeater.cc:503] Master 127.0.97.62:35543 was elected leader, sending a full tablet report...
I20260925 10:32:10.186801 2019 ts_tablet_manager.cc:1428] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260925 10:32:10.260960 2021 raft_consensus.cc:482] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:32:10.261253 2021 raft_consensus.cc:504] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } }
I20260925 10:32:10.262792 2021 leader_election.cc:288] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3f05340707d4837a46fa30eca0a59e5 (127.0.97.1:39545), e2c1759857fa4822a2318abd541d8258 (127.0.97.3:34967)
I20260925 10:32:10.268864 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.263123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48505 (local address 127.0.97.1:39545)
0925 10:32:10.263334 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:10.263343 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:10.263813 (+ 470us) server_negotiation.cc:407] Connection header received
0925 10:32:10.264026 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:10.264035 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:10.264138 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:10.264223 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:10.264791 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.265454 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.267198 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.267940 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.268139 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.268241 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:10.268505 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 10:32:10.268622 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:32:10.269006 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.263121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39545 (local address 127.0.97.2:48505)
0925 10:32:10.263696 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:10.263711 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:10.263874 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:10.264302 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:10.264306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:10.264330 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:10.264659 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.264667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.265580 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.265586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.267055 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.267066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.268194 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.268200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.268353 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.268376 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:32:10.268481 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 10:32:10.268598 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":332,"threads_started":1}
I20260925 10:32:10.269716 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.263968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52963 (local address 127.0.97.3:34967)
0925 10:32:10.264156 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:10.264164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:10.264479 (+ 315us) server_negotiation.cc:407] Connection header received
0925 10:32:10.264692 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:10.264697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:10.264756 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:10.264842 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:10.265353 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.266008 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.267735 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.268626 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.268768 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.268873 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:10.269372 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 10:32:10.269482 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:32:10.269742 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.263845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34967 (local address 127.0.97.2:52963)
0925 10:32:10.264379 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:10.264397 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:10.264561 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:10.264863 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:10.264867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:10.264888 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:10.265201 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.265208 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.266209 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.266213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.267597 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.267607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.268839 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.268845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.269121 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.269163 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 10:32:10.269318 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 10:32:10.269446 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":252,"threads_started":1}
I20260925 10:32:10.270514 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a633b2883464db383e6edc4059553a3" candidate_uuid: "18ed0d768ca64db88c485109e83ee0ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f05340707d4837a46fa30eca0a59e5" is_pre_election: true
I20260925 10:32:10.270956 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a633b2883464db383e6edc4059553a3" candidate_uuid: "18ed0d768ca64db88c485109e83ee0ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c1759857fa4822a2318abd541d8258" is_pre_election: true
I20260925 10:32:10.271100 1673 raft_consensus.cc:2454] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18ed0d768ca64db88c485109e83ee0ac in term 0.
I20260925 10:32:10.271450 1946 raft_consensus.cc:2454] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18ed0d768ca64db88c485109e83ee0ac in term 0.
I20260925 10:32:10.271698 1738 leader_election.cc:302] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ed0d768ca64db88c485109e83ee0ac, a3f05340707d4837a46fa30eca0a59e5; no voters:
I20260925 10:32:10.272099 2021 raft_consensus.cc:2781] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:32:10.272228 2021 raft_consensus.cc:482] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:32:10.272333 2021 raft_consensus.cc:3037] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:10.274533 2021 raft_consensus.cc:504] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } }
I20260925 10:32:10.275065 2021 leader_election.cc:288] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [CANDIDATE]: Term 1 election: Requested vote from peers a3f05340707d4837a46fa30eca0a59e5 (127.0.97.1:39545), e2c1759857fa4822a2318abd541d8258 (127.0.97.3:34967)
I20260925 10:32:10.275470 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a633b2883464db383e6edc4059553a3" candidate_uuid: "18ed0d768ca64db88c485109e83ee0ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f05340707d4837a46fa30eca0a59e5"
I20260925 10:32:10.275650 1673 raft_consensus.cc:3037] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:10.275627 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a633b2883464db383e6edc4059553a3" candidate_uuid: "18ed0d768ca64db88c485109e83ee0ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c1759857fa4822a2318abd541d8258"
I20260925 10:32:10.275818 1946 raft_consensus.cc:3037] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:10.277743 1673 raft_consensus.cc:2454] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18ed0d768ca64db88c485109e83ee0ac in term 1.
I20260925 10:32:10.277920 1946 raft_consensus.cc:2454] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18ed0d768ca64db88c485109e83ee0ac in term 1.
I20260925 10:32:10.278115 1738 leader_election.cc:302] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ed0d768ca64db88c485109e83ee0ac, a3f05340707d4837a46fa30eca0a59e5; no voters:
I20260925 10:32:10.278435 2021 raft_consensus.cc:2781] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:10.278857 2021 raft_consensus.cc:686] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [term 1 LEADER]: Becoming Leader. State: Replica: 18ed0d768ca64db88c485109e83ee0ac, State: Running, Role: LEADER
I20260925 10:32:10.279281 2021 consensus_queue.cc:237] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } }
I20260925 10:32:10.283795 1511 catalog_manager.cc:5495] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 18ed0d768ca64db88c485109e83ee0ac (127.0.97.2). New cstate: current_term: 1 leader_uuid: "18ed0d768ca64db88c485109e83ee0ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } health_report { overall_health: HEALTHY } } }
W20260925 10:32:10.299749 1997 tablet.cc:2369] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:32:10.306943 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.302076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55250 (local address 127.0.97.2:35825)
0925 10:32:10.302246 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:10.302251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:10.302451 (+ 200us) server_negotiation.cc:407] Connection header received
0925 10:32:10.302687 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:10.302691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:10.302759 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:10.302869 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:32:10.303175 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.303856 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.305090 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.305452 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.305536 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.305973 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:32:10.306502 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:32:10.306612 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:10.306704 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 10:32:10.306759 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 10:32:10.308950 1858 tablet.cc:2369] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:32:10.314201 1946 raft_consensus.cc:1261] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258 [term 1 FOLLOWER]: Refusing update from remote peer 18ed0d768ca64db88c485109e83ee0ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:10.314215 1673 raft_consensus.cc:1261] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5 [term 1 FOLLOWER]: Refusing update from remote peer 18ed0d768ca64db88c485109e83ee0ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:10.315042 2026 consensus_queue.cc:1035] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:10.315393 2021 consensus_queue.cc:1035] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:10.327162 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333201224957890560 to 7333201224827301888. Current Snapshot: MvccSnapshot[applied={T|T < 7333201224957890560}]
I20260925 10:32:10.350111 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:10.350384 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:10.353466 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee: Sending DeleteTablet for 3 replicas of tablet 9a633b2883464db383e6edc4059553a3
I20260925 10:32:10.354558 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a633b2883464db383e6edc4059553a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55236
I20260925 10:32:10.354835 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a633b2883464db383e6edc4059553a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36402
I20260925 10:32:10.355484 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a633b2883464db383e6edc4059553a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37496
I20260925 10:32:10.355922 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:32:10.356892 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:32:10.359622 2042 tablet_replica.cc:330] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258: stopping tablet replica
I20260925 10:32:10.360168 2042 raft_consensus.cc:2229] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:10.362591 2042 raft_consensus.cc:2258] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:10.363822 2040 tablet_replica.cc:330] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac: stopping tablet replica
I20260925 10:32:10.364243 2040 raft_consensus.cc:2229] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:32:10.365008 2042 ts_tablet_manager.cc:1905] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:10.366228 1647 tablet_service.cc:1467] Processing CreateTablet for tablet bc699dcf35e84624922c184b59ae0c17 (DEFAULT_TABLE table=test-table [id=56e091ca8fae4b50af84c2eac322e94a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:10.366497 2041 tablet_replica.cc:330] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5: stopping tablet replica
I20260925 10:32:10.366485 1926 tablet_service.cc:1467] Processing CreateTablet for tablet bc699dcf35e84624922c184b59ae0c17 (DEFAULT_TABLE table=test-table [id=56e091ca8fae4b50af84c2eac322e94a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:10.366843 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc699dcf35e84624922c184b59ae0c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:10.366916 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc699dcf35e84624922c184b59ae0c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:10.366870 2041 raft_consensus.cc:2229] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:10.368372 2041 raft_consensus.cc:2258] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:10.372895 2042 ts_tablet_manager.cc:1918] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:32:10.373097 2042 log.cc:1196] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/9a633b2883464db383e6edc4059553a3
I20260925 10:32:10.373278 2019 tablet_bootstrap.cc:492] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258: Bootstrap starting.
I20260925 10:32:10.373780 2042 ts_tablet_manager.cc:1939] T 9a633b2883464db383e6edc4059553a3 P e2c1759857fa4822a2318abd541d8258: Deleting consensus metadata
I20260925 10:32:10.374557 1787 tablet_service.cc:1467] Processing CreateTablet for tablet bc699dcf35e84624922c184b59ae0c17 (DEFAULT_TABLE table=test-table [id=56e091ca8fae4b50af84c2eac322e94a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:10.374763 2016 tablet_bootstrap.cc:492] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5: Bootstrap starting.
I20260925 10:32:10.375789 1495 catalog_manager.cc:4841] TS e2c1759857fa4822a2318abd541d8258 (127.0.97.3:34967): tablet 9a633b2883464db383e6edc4059553a3 (table test-table [id=55ff213cdd4e40f4a521257fe6eb71ad]) successfully deleted
I20260925 10:32:10.376210 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc699dcf35e84624922c184b59ae0c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:10.376235 2016 tablet_bootstrap.cc:654] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:10.377570 2041 ts_tablet_manager.cc:1905] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:10.377954 2019 tablet_bootstrap.cc:654] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:10.384892 2041 ts_tablet_manager.cc:1918] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:32:10.385056 2041 log.cc:1196] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/9a633b2883464db383e6edc4059553a3
I20260925 10:32:10.385231 2015 tablet_bootstrap.cc:492] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac: Bootstrap starting.
I20260925 10:32:10.385609 2041 ts_tablet_manager.cc:1939] T 9a633b2883464db383e6edc4059553a3 P a3f05340707d4837a46fa30eca0a59e5: Deleting consensus metadata
I20260925 10:32:10.384883 2040 raft_consensus.cc:2258] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:10.387207 2015 tablet_bootstrap.cc:654] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:10.387558 2016 tablet_bootstrap.cc:492] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5: No bootstrap required, opened a new log
I20260925 10:32:10.387691 2016 ts_tablet_manager.cc:1397] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20260925 10:32:10.388479 2016 raft_consensus.cc:348] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.388715 2016 raft_consensus.cc:374] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:10.388813 2016 raft_consensus.cc:729] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f05340707d4837a46fa30eca0a59e5, State: Initialized, Role: FOLLOWER
I20260925 10:32:10.389128 2040 ts_tablet_manager.cc:1905] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:10.389065 2016 consensus_queue.cc:260] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.389632 2016 ts_tablet_manager.cc:1428] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:32:10.390784 1495 catalog_manager.cc:4841] TS a3f05340707d4837a46fa30eca0a59e5 (127.0.97.1:39545): tablet 9a633b2883464db383e6edc4059553a3 (table test-table [id=55ff213cdd4e40f4a521257fe6eb71ad]) successfully deleted
I20260925 10:32:10.395704 2019 tablet_bootstrap.cc:492] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258: No bootstrap required, opened a new log
I20260925 10:32:10.395851 2019 ts_tablet_manager.cc:1397] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.005s
I20260925 10:32:10.395946 2015 tablet_bootstrap.cc:492] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac: No bootstrap required, opened a new log
I20260925 10:32:10.396076 2015 ts_tablet_manager.cc:1397] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260925 10:32:10.396633 2019 raft_consensus.cc:348] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.396875 2019 raft_consensus.cc:374] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:10.396831 2015 raft_consensus.cc:348] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.396974 2019 raft_consensus.cc:729] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c1759857fa4822a2318abd541d8258, State: Initialized, Role: FOLLOWER
I20260925 10:32:10.397053 2015 raft_consensus.cc:374] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:10.397217 2015 raft_consensus.cc:729] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ed0d768ca64db88c485109e83ee0ac, State: Initialized, Role: FOLLOWER
I20260925 10:32:10.397219 2019 consensus_queue.cc:260] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.397789 2019 ts_tablet_manager.cc:1428] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 10:32:10.397481 2015 consensus_queue.cc:260] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.400904 2040 ts_tablet_manager.cc:1918] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:32:10.401125 2040 log.cc:1196] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/9a633b2883464db383e6edc4059553a3
I20260925 10:32:10.401674 2040 ts_tablet_manager.cc:1939] T 9a633b2883464db383e6edc4059553a3 P 18ed0d768ca64db88c485109e83ee0ac: Deleting consensus metadata
I20260925 10:32:10.401885 2015 ts_tablet_manager.cc:1428] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260925 10:32:10.403365 1496 catalog_manager.cc:4841] TS 18ed0d768ca64db88c485109e83ee0ac (127.0.97.2:35825): tablet 9a633b2883464db383e6edc4059553a3 (table test-table [id=55ff213cdd4e40f4a521257fe6eb71ad]) successfully deleted
I20260925 10:32:10.581475 2021 raft_consensus.cc:482] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:32:10.581681 2021 raft_consensus.cc:504] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.582432 2021 leader_election.cc:288] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3f05340707d4837a46fa30eca0a59e5 (127.0.97.1:39545), e2c1759857fa4822a2318abd541d8258 (127.0.97.3:34967)
I20260925 10:32:10.583071 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc699dcf35e84624922c184b59ae0c17" candidate_uuid: "18ed0d768ca64db88c485109e83ee0ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f05340707d4837a46fa30eca0a59e5" is_pre_election: true
I20260925 10:32:10.583129 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc699dcf35e84624922c184b59ae0c17" candidate_uuid: "18ed0d768ca64db88c485109e83ee0ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c1759857fa4822a2318abd541d8258" is_pre_election: true
I20260925 10:32:10.583335 1673 raft_consensus.cc:2454] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18ed0d768ca64db88c485109e83ee0ac in term 0.
I20260925 10:32:10.583395 1946 raft_consensus.cc:2454] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18ed0d768ca64db88c485109e83ee0ac in term 0.
I20260925 10:32:10.583766 1738 leader_election.cc:302] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ed0d768ca64db88c485109e83ee0ac, a3f05340707d4837a46fa30eca0a59e5; no voters:
W20260925 10:32:10.584080 1719 tablet.cc:2369] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:32:10.584283 2021 raft_consensus.cc:2781] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:32:10.584403 2021 raft_consensus.cc:482] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:32:10.584908 2021 raft_consensus.cc:3037] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:10.588160 2021 raft_consensus.cc:504] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.588771 2021 leader_election.cc:288] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [CANDIDATE]: Term 1 election: Requested vote from peers a3f05340707d4837a46fa30eca0a59e5 (127.0.97.1:39545), e2c1759857fa4822a2318abd541d8258 (127.0.97.3:34967)
I20260925 10:32:10.589224 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc699dcf35e84624922c184b59ae0c17" candidate_uuid: "18ed0d768ca64db88c485109e83ee0ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f05340707d4837a46fa30eca0a59e5"
I20260925 10:32:10.589316 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc699dcf35e84624922c184b59ae0c17" candidate_uuid: "18ed0d768ca64db88c485109e83ee0ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c1759857fa4822a2318abd541d8258"
I20260925 10:32:10.589429 1673 raft_consensus.cc:3037] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:10.589495 1946 raft_consensus.cc:3037] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:10.592250 1673 raft_consensus.cc:2454] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18ed0d768ca64db88c485109e83ee0ac in term 1.
I20260925 10:32:10.592250 1946 raft_consensus.cc:2454] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18ed0d768ca64db88c485109e83ee0ac in term 1.
I20260925 10:32:10.592644 1738 leader_election.cc:302] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ed0d768ca64db88c485109e83ee0ac, a3f05340707d4837a46fa30eca0a59e5; no voters:
I20260925 10:32:10.592967 2021 raft_consensus.cc:2781] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:10.593508 2021 raft_consensus.cc:686] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [term 1 LEADER]: Becoming Leader. State: Replica: 18ed0d768ca64db88c485109e83ee0ac, State: Running, Role: LEADER
I20260925 10:32:10.593767 2021 consensus_queue.cc:237] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.596580 1511 catalog_manager.cc:5495] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 18ed0d768ca64db88c485109e83ee0ac (127.0.97.2). New cstate: current_term: 1 leader_uuid: "18ed0d768ca64db88c485109e83ee0ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } health_report { overall_health: UNKNOWN } } }
Sep 25 10:32:10 dist-test-slave-dxdd krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:10.647959 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.632581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35543 (local address 127.0.0.1:43906)
0925 10:32:10.633918 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:10.633936 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:10.634058 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:10.634452 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:10.634455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:10.635110 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:10.635363 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.635372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.636867 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.636870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.637774 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.637784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.638743 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.638747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.638999 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.639705 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:10.639721 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:10.642010 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:10.645717 (+ 3707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:10.645725 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:10.645731 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:10.646099 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:10.646517 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:10.646521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:10.646523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:10.646645 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:10.647255 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:10.647261 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:10.647407 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 10:32:10.647522 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 10:32:10.647707 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1197,"thread_start_us":547,"threads_started":1}
I20260925 10:32:10.648305 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.632581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.0.97.62:35543)
0925 10:32:10.633643 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:10.633649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:10.634046 (+ 397us) server_negotiation.cc:407] Connection header received
0925 10:32:10.634171 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:10.634178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:10.634255 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:10.634415 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:10.635528 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.636732 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.637912 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.638600 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.638743 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.642218 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:10.642224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:10.642229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:10.642265 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:10.645584 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:10.646231 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:10.646237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:10.646240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:10.646408 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:10.646762 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:10.646768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:10.646771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:10.647126 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:10.647262 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:10.647654 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 10:32:10.647866 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":305,"threads_started":1}
I20260925 10:32:10.653185 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43906:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:32:10.654163 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:32:10.660624 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 711bb21414fe41d087ae7e1ead055d97 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a8cd3d648326483cb9f40ef1a669043d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:32:10.660552 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 711bb21414fe41d087ae7e1ead055d97 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a8cd3d648326483cb9f40ef1a669043d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:32:10.661154 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711bb21414fe41d087ae7e1ead055d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:10.661158 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711bb21414fe41d087ae7e1ead055d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:10.666440 2015 tablet_bootstrap.cc:492] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac: Bootstrap starting.
I20260925 10:32:10.667589 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 711bb21414fe41d087ae7e1ead055d97 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a8cd3d648326483cb9f40ef1a669043d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:32:10.668020 2016 tablet_bootstrap.cc:492] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5: Bootstrap starting.
I20260925 10:32:10.668120 2015 tablet_bootstrap.cc:654] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:10.668130 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711bb21414fe41d087ae7e1ead055d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:10.670002 2016 tablet_bootstrap.cc:654] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:10.676774 2019 tablet_bootstrap.cc:492] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258: Bootstrap starting.
I20260925 10:32:10.678251 2019 tablet_bootstrap.cc:654] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:10.680517 2019 tablet_bootstrap.cc:492] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258: No bootstrap required, opened a new log
I20260925 10:32:10.680699 2019 ts_tablet_manager.cc:1397] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 10:32:10.681423 2019 raft_consensus.cc:348] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.681630 2019 raft_consensus.cc:374] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:10.681703 2019 raft_consensus.cc:729] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c1759857fa4822a2318abd541d8258, State: Initialized, Role: FOLLOWER
I20260925 10:32:10.681742 2016 tablet_bootstrap.cc:492] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5: No bootstrap required, opened a new log
I20260925 10:32:10.681892 2016 ts_tablet_manager.cc:1397] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260925 10:32:10.681967 2019 consensus_queue.cc:260] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.682513 2019 ts_tablet_manager.cc:1428] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:32:10.682446 2023 tablet_replica.cc:439] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } }
I20260925 10:32:10.682765 2023 tablet_replica.cc:442] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:32:10.683620 2016 raft_consensus.cc:348] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.683877 2016 raft_consensus.cc:374] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:10.683957 2016 raft_consensus.cc:729] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f05340707d4837a46fa30eca0a59e5, State: Initialized, Role: FOLLOWER
I20260925 10:32:10.684216 2015 tablet_bootstrap.cc:492] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac: No bootstrap required, opened a new log
I20260925 10:32:10.684393 2015 ts_tablet_manager.cc:1397] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.007s
I20260925 10:32:10.685029 2016 consensus_queue.cc:260] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.685092 2015 raft_consensus.cc:348] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.685427 2015 raft_consensus.cc:374] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:10.685613 2022 tablet_replica.cc:439] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } }
I20260925 10:32:10.685932 2015 raft_consensus.cc:729] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ed0d768ca64db88c485109e83ee0ac, State: Initialized, Role: FOLLOWER
I20260925 10:32:10.685968 2022 tablet_replica.cc:442] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:32:10.686378 2015 consensus_queue.cc:260] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.687578 2016 ts_tablet_manager.cc:1428] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260925 10:32:10.687338 2021 tablet_replica.cc:439] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } }
I20260925 10:32:10.687842 2021 tablet_replica.cc:442] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:32:10.688491 2015 ts_tablet_manager.cc:1428] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 10:32:10.898208 2022 raft_consensus.cc:482] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:32:10.898476 2022 raft_consensus.cc:504] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.899869 2022 leader_election.cc:288] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18ed0d768ca64db88c485109e83ee0ac (127.0.97.2:35825), e2c1759857fa4822a2318abd541d8258 (127.0.97.3:34967)
I20260925 10:32:10.916869 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.900257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44923 (local address 127.0.97.3:34967)
0925 10:32:10.911608 (+ 11351us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:10.911612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:10.911627 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:32:10.911690 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:10.911693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:10.911747 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:10.911861 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:10.912363 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.913130 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.914992 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.915910 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.916048 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.916147 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:10.916446 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 10:32:10.916575 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11263,"thread_start_us":11171,"threads_started":1}
I20260925 10:32:10.916956 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.900193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34967 (local address 127.0.97.1:44923)
0925 10:32:10.910997 (+ 10804us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:10.911022 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:10.911178 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:10.911879 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:10.911883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:10.911908 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:10.912182 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.912191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.913270 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.913280 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.914828 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.914841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.916060 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.916065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.916233 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.916274 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 10:32:10.916415 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 10:32:10.916565 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10694,"mutex_wait_us":41,"thread_start_us":11100,"threads_started":1}
I20260925 10:32:10.917397 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.900265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60255 (local address 127.0.97.2:35825)
0925 10:32:10.911292 (+ 11027us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:10.911302 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:10.911692 (+ 390us) server_negotiation.cc:407] Connection header received
0925 10:32:10.911893 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:10.911897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:10.911952 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:10.912057 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:10.912643 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.913320 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.915447 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.916203 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.916381 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.916484 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:10.916832 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 10:32:10.916960 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10893,"thread_start_us":10823,"threads_started":1}
I20260925 10:32:10.917572 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.900294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35825 (local address 127.0.97.1:60255)
0925 10:32:10.911587 (+ 11293us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:10.911601 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:10.911743 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:10.912074 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:10.912078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:10.912105 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:10.912505 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.912514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.913543 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.913551 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.915294 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.915307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.916432 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.916438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.916618 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.916699 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 10:32:10.916797 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 10:32:10.916920 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11185,"mutex_wait_us":84,"thread_start_us":10515,"threads_started":1}
I20260925 10:32:10.917598 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711bb21414fe41d087ae7e1ead055d97" candidate_uuid: "a3f05340707d4837a46fa30eca0a59e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c1759857fa4822a2318abd541d8258" is_pre_election: true
I20260925 10:32:10.917845 1946 raft_consensus.cc:2454] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3f05340707d4837a46fa30eca0a59e5 in term 0.
I20260925 10:32:10.918785 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711bb21414fe41d087ae7e1ead055d97" candidate_uuid: "a3f05340707d4837a46fa30eca0a59e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18ed0d768ca64db88c485109e83ee0ac" is_pre_election: true
I20260925 10:32:10.918874 1598 leader_election.cc:302] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3f05340707d4837a46fa30eca0a59e5, e2c1759857fa4822a2318abd541d8258; no voters:
I20260925 10:32:10.919174 1807 raft_consensus.cc:2454] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3f05340707d4837a46fa30eca0a59e5 in term 0.
I20260925 10:32:10.919238 2022 raft_consensus.cc:2781] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:32:10.919366 2022 raft_consensus.cc:482] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:32:10.919480 2022 raft_consensus.cc:3037] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:10.921563 2022 raft_consensus.cc:504] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.922127 2022 leader_election.cc:288] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [CANDIDATE]: Term 1 election: Requested vote from peers 18ed0d768ca64db88c485109e83ee0ac (127.0.97.2:35825), e2c1759857fa4822a2318abd541d8258 (127.0.97.3:34967)
I20260925 10:32:10.922485 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711bb21414fe41d087ae7e1ead055d97" candidate_uuid: "a3f05340707d4837a46fa30eca0a59e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18ed0d768ca64db88c485109e83ee0ac"
I20260925 10:32:10.922677 1807 raft_consensus.cc:3037] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:10.922717 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711bb21414fe41d087ae7e1ead055d97" candidate_uuid: "a3f05340707d4837a46fa30eca0a59e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c1759857fa4822a2318abd541d8258"
I20260925 10:32:10.922968 1946 raft_consensus.cc:3037] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:10.924808 1807 raft_consensus.cc:2454] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3f05340707d4837a46fa30eca0a59e5 in term 1.
I20260925 10:32:10.924906 1946 raft_consensus.cc:2454] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3f05340707d4837a46fa30eca0a59e5 in term 1.
I20260925 10:32:10.925227 1599 leader_election.cc:302] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ed0d768ca64db88c485109e83ee0ac, a3f05340707d4837a46fa30eca0a59e5; no voters:
I20260925 10:32:10.925539 2022 raft_consensus.cc:2781] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:10.925931 2022 raft_consensus.cc:686] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [term 1 LEADER]: Becoming Leader. State: Replica: a3f05340707d4837a46fa30eca0a59e5, State: Running, Role: LEADER
I20260925 10:32:10.926344 2022 consensus_queue.cc:237] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } }
I20260925 10:32:10.928097 2061 tablet_replica.cc:439] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5: TxnStatusTablet state changed. Reason: New leader a3f05340707d4837a46fa30eca0a59e5. Latest consensus state: current_term: 1 leader_uuid: "a3f05340707d4837a46fa30eca0a59e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } }
I20260925 10:32:10.928536 2061 tablet_replica.cc:442] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:32:10.929328 2063 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:32:10.929543 2063 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:32:10.929993 1510 catalog_manager.cc:5495] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3f05340707d4837a46fa30eca0a59e5 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a3f05340707d4837a46fa30eca0a59e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } health_report { overall_health: UNKNOWN } } }
I20260925 10:32:10.986752 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.979505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39545 (local address 127.0.0.1:36406)
0925 10:32:10.979788 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:10.979808 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:10.979957 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:10.981206 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:10.981210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:10.981220 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:32:10.981369 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.981376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.982929 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.982933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.983881 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:10.983891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.984919 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.984923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:10.985049 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.985134 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:32:10.986293 (+ 1159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:32:10.986345 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 10:32:10.986424 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:32:10.986530 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 10:32:10.986969 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:10.979675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36406 (local address 127.0.97.1:39545)
0925 10:32:10.980774 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:10.980780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:10.980804 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:32:10.980902 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:10.980908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:10.981011 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:10.981162 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:32:10.981509 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.982805 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.984052 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:10.984797 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:10.984923 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:10.985389 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:32:10.986128 (+ 739us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:32:10.986319 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:10.986480 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 10:32:10.986561 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":379,"threads_started":1}
I20260925 10:32:10.991161 1946 raft_consensus.cc:1261] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258 [term 1 FOLLOWER]: Refusing update from remote peer a3f05340707d4837a46fa30eca0a59e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:10.991573 1807 raft_consensus.cc:1261] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac [term 1 FOLLOWER]: Refusing update from remote peer a3f05340707d4837a46fa30eca0a59e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:10.991977 2022 consensus_queue.cc:1035] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:10.992297 2061 consensus_queue.cc:1035] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:10.999907 2021 tablet_replica.cc:439] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac: TxnStatusTablet state changed. Reason: New leader a3f05340707d4837a46fa30eca0a59e5. Latest consensus state: current_term: 1 leader_uuid: "a3f05340707d4837a46fa30eca0a59e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } }
I20260925 10:32:11.000175 2021 tablet_replica.cc:442] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:32:11.002795 2021 consensus_queue.cc:1035] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:11.003418 2061 tablet_replica.cc:439] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3f05340707d4837a46fa30eca0a59e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } }
I20260925 10:32:11.003652 2061 tablet_replica.cc:442] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:32:11.007424 2023 tablet_replica.cc:439] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258: TxnStatusTablet state changed. Reason: New leader a3f05340707d4837a46fa30eca0a59e5. Latest consensus state: current_term: 1 leader_uuid: "a3f05340707d4837a46fa30eca0a59e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } }
I20260925 10:32:11.007680 2023 tablet_replica.cc:442] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:32:11.009179 2022 tablet_replica.cc:439] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3f05340707d4837a46fa30eca0a59e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } }
I20260925 10:32:11.009431 2022 tablet_replica.cc:442] T 711bb21414fe41d087ae7e1ead055d97 P a3f05340707d4837a46fa30eca0a59e5: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:32:11.010835 2021 tablet_replica.cc:439] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3f05340707d4837a46fa30eca0a59e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } }
I20260925 10:32:11.011034 2021 tablet_replica.cc:442] T 711bb21414fe41d087ae7e1ead055d97 P 18ed0d768ca64db88c485109e83ee0ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:32:11.022715 2023 tablet_replica.cc:439] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3f05340707d4837a46fa30eca0a59e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 } } peers { permanent_uuid: "18ed0d768ca64db88c485109e83ee0ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35825 } } peers { permanent_uuid: "e2c1759857fa4822a2318abd541d8258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34967 } } }
I20260925 10:32:11.022949 2023 tablet_replica.cc:442] T 711bb21414fe41d087ae7e1ead055d97 P e2c1759857fa4822a2318abd541d8258: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:32:11.023654 2021 consensus_queue.cc:1035] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3f05340707d4837a46fa30eca0a59e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:11.042647 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:11.036448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33437 (local address 127.0.97.1:39545)
0925 10:32:11.036613 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:11.036620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:11.037175 (+ 555us) server_negotiation.cc:407] Connection header received
0925 10:32:11.037327 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:11.037334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:11.037421 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:11.037523 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:32:11.037903 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.039109 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.040518 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.041187 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.041313 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:11.041883 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:32:11.041999 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:32:11.042056 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:11.042263 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 10:32:11.042363 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:32:11.043509 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:11.036436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39545 (local address 127.0.97.2:33437)
0925 10:32:11.037082 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:11.037102 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:11.037219 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:11.037534 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:11.037537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:11.037546 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:32:11.037781 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:11.037788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.039235 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.039239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:11.040391 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:11.040402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.041432 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.041438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:11.041604 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:11.041674 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:32:11.042132 (+ 458us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:32:11.042141 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:32:11.043017 (+ 876us) client_negotiation.cc:240] Negotiation successful
0925 10:32:11.043155 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":344,"threads_started":1}
I20260925 10:32:11.112053 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:11.099603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41441 (local address 127.0.97.2:35825)
0925 10:32:11.099805 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:11.099813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:11.100585 (+ 772us) server_negotiation.cc:407] Connection header received
0925 10:32:11.101015 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:11.101023 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:11.101154 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:11.101226 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:32:11.103914 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.104998 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.109795 (+ 4797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.110278 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.110452 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:11.111060 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:32:11.111190 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:32:11.111254 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:11.111776 (+ 522us) server_negotiation.cc:299] Negotiation successful
0925 10:32:11.111862 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:32:11.113770 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:11.099603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35825 (local address 127.0.97.1:41441)
0925 10:32:11.100466 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:11.100490 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:11.100827 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:11.103355 (+ 2528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:11.103360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:11.103373 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:32:11.103749 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:11.103759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.105185 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.105190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:11.109639 (+ 4449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:11.109651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.110492 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.110500 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:11.110637 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:11.110687 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:32:11.111621 (+ 934us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:32:11.111632 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 10:32:11.112617 (+ 985us) client_negotiation.cc:240] Negotiation successful
0925 10:32:11.112904 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":414,"threads_started":1}
I20260925 10:32:11.151943 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:11.152166 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:11.154899 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 914c2911f1014c30bac107166544e0ee: Sending DeleteTablet for 3 replicas of tablet bc699dcf35e84624922c184b59ae0c17
I20260925 10:32:11.155650 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet bc699dcf35e84624922c184b59ae0c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55236
I20260925 10:32:11.155748 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet bc699dcf35e84624922c184b59ae0c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36402
I20260925 10:32:11.155776 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet bc699dcf35e84624922c184b59ae0c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37496
I20260925 10:32:11.167464 2087 tablet_replica.cc:330] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac: stopping tablet replica
I20260925 10:32:11.167840 2088 tablet_replica.cc:330] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5: stopping tablet replica
I20260925 10:32:11.167809 2087 raft_consensus.cc:2229] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:32:11.168066 2088 raft_consensus.cc:2229] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:11.168392 2087 raft_consensus.cc:2258] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:11.169090 2088 raft_consensus.cc:2258] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:11.169683 2087 ts_tablet_manager.cc:1905] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:11.170194 2088 ts_tablet_manager.cc:1905] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:11.177654 2088 ts_tablet_manager.cc:1918] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:32:11.178375 2089 tablet_replica.cc:330] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258: stopping tablet replica
I20260925 10:32:11.178567 2089 raft_consensus.cc:2229] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:11.178810 2089 raft_consensus.cc:2258] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:11.178884 2087 ts_tablet_manager.cc:1918] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:32:11.179009 2087 log.cc:1196] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/bc699dcf35e84624922c184b59ae0c17
I20260925 10:32:11.179430 2087 ts_tablet_manager.cc:1939] T bc699dcf35e84624922c184b59ae0c17 P 18ed0d768ca64db88c485109e83ee0ac: Deleting consensus metadata
I20260925 10:32:11.179805 2089 ts_tablet_manager.cc:1905] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:11.180404 1496 catalog_manager.cc:4841] TS 18ed0d768ca64db88c485109e83ee0ac (127.0.97.2:35825): tablet bc699dcf35e84624922c184b59ae0c17 (table test-table [id=56e091ca8fae4b50af84c2eac322e94a]) successfully deleted
I20260925 10:32:11.180699 2088 log.cc:1196] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/bc699dcf35e84624922c184b59ae0c17
I20260925 10:32:11.181159 2088 ts_tablet_manager.cc:1939] T bc699dcf35e84624922c184b59ae0c17 P a3f05340707d4837a46fa30eca0a59e5: Deleting consensus metadata
I20260925 10:32:11.182132 1495 catalog_manager.cc:4841] TS a3f05340707d4837a46fa30eca0a59e5 (127.0.97.1:39545): tablet bc699dcf35e84624922c184b59ae0c17 (table test-table [id=56e091ca8fae4b50af84c2eac322e94a]) successfully deleted
I20260925 10:32:11.186684 2089 ts_tablet_manager.cc:1918] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:32:11.186815 2089 log.cc:1196] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/bc699dcf35e84624922c184b59ae0c17
I20260925 10:32:11.187204 2089 ts_tablet_manager.cc:1939] T bc699dcf35e84624922c184b59ae0c17 P e2c1759857fa4822a2318abd541d8258: Deleting consensus metadata
I20260925 10:32:11.188659 1495 catalog_manager.cc:4841] TS e2c1759857fa4822a2318abd541d8258 (127.0.97.3:34967): tablet bc699dcf35e84624922c184b59ae0c17 (table test-table [id=56e091ca8fae4b50af84c2eac322e94a]) successfully deleted
Sep 25 10:32:11 dist-test-slave-dxdd krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:32:11.195825 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:11.168542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.0.97.1:39545)
0925 10:32:11.168735 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:11.168741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:11.174800 (+ 6059us) server_negotiation.cc:407] Connection header received
0925 10:32:11.175056 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:11.175061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:11.175150 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:11.175229 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:11.181988 (+ 6759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.182923 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.184498 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.185046 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.185207 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:11.190102 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:11.190117 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:11.190135 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:11.190164 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:11.193182 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:11.193839 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:11.193846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:11.193848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:11.193986 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:11.194386 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:11.194389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:11.194391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:11.194732 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:11.195068 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:11.195299 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 10:32:11.195500 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 10:32:11.196548 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36410
I20260925 10:32:11.214619 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:11.199143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.0.97.62:35543)
0925 10:32:11.202716 (+ 3573us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:11.202722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:11.202739 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:32:11.202797 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:11.202801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:11.202871 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:11.202997 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:11.204150 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.204907 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.206490 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.206901 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.207033 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:11.209206 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:11.209211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:11.209214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:11.209244 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:11.211904 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:11.212502 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:11.212507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:11.212510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:11.212694 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:11.213070 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:11.213075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:11.213080 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:11.213419 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:11.213688 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:11.213989 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 10:32:11.214208 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3463,"thread_start_us":3301,"threads_started":1}
W20260925 10:32:11.215247 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43922
I20260925 10:32:11.217008 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 1582
I20260925 10:32:11.230543 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:11.218071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36416 (local address 127.0.97.1:39545)
0925 10:32:11.218314 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:11.218341 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:11.218673 (+ 332us) server_negotiation.cc:407] Connection header received
0925 10:32:11.218860 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:11.218864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:11.218921 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:11.219030 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:11.220041 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.220707 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.221832 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.222282 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.222441 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:11.225081 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:11.225085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:11.225088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:11.225114 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:11.227839 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:11.228488 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:11.228498 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:11.228500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:11.228649 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:11.228989 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:11.228995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:11.228999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:11.229317 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:11.229915 (+ 598us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:11.230045 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 10:32:11.230248 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 10:32:11.231104 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36416
W20260925 10:32:11.231612 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:32:11.231786 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 1582
I20260925 10:32:11.246671 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 1722
Sep 25 10:32:11 dist-test-slave-dxdd krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:32:11.260025 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:11.247250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.0.97.2:35825)
0925 10:32:11.247417 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:11.247425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:11.247485 (+ 60us) server_negotiation.cc:407] Connection header received
0925 10:32:11.247665 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:11.247671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:11.247736 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:11.247860 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:11.248897 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.249566 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.250873 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.251256 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.251375 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:11.255156 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:11.255167 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:11.255181 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:11.255216 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:11.257868 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:11.258493 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:11.258499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:11.258502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:11.258619 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:11.258919 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:11.258922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:11.258924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:11.259233 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:11.259437 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:11.259645 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 10:32:11.259785 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 10:32:11.260607 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55262
W20260925 10:32:11.261148 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:32:11.261303 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 1722
I20260925 10:32:11.281913 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 1861
Sep 25 10:32:11 dist-test-slave-dxdd krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:32:11.295333 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:11.282638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.0.97.3:34967)
0925 10:32:11.282917 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:11.282924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:11.282944 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:32:11.283048 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:11.283054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:11.283147 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:11.283279 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:11.284533 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.285173 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.286229 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:11.286752 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:11.286843 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:11.290589 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:11.290601 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:11.290616 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:11.290644 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:11.293197 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:11.293763 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:11.293769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:11.293771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:11.293880 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:11.294153 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:11.294156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:11.294158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:11.294584 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:11.294792 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:11.294967 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 10:32:11.295107 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
W20260925 10:32:11.295984 1991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37508
W20260925 10:32:11.296590 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:32:11.296697 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 1861
I20260925 10:32:11.312940 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 1479
I20260925 10:32:11.313886 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:11.422969 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 1479
2026-09-25T10:32:11Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4590 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:32:11 dist-test-slave-dxdd krb5kdc[2099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:32:11 dist-test-slave-dxdd krb5kdc[2099](info): set up 2 sockets
Sep 25 10:32:11 dist-test-slave-dxdd krb5kdc[2099](info): commencing operation
krb5kdc: starting...
W20260925 10:32:12.498449 388 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:32:12 dist-test-slave-dxdd krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:32:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:32:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:32:12.673728 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34799
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:12.980141 2112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:12.980454 2112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:12.980614 2112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:12.988421 2112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:12.988540 2112 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:32:12.988631 2112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:12.988737 2112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:12.988827 2112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:12.988910 2112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:32:12.998654 2112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34799
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:12.999863 2112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:13.001139 2112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:13.012391 2117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:13.012444 2120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:13.012470 2118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:13.013579 2112 server_base.cc:1034] running on GCE node
I20260925 10:32:13.014281 2112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:13.015568 2112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:13.016782 2112 hybrid_clock.cc:648] HybridClock initialized: now 1790332333016724 us; error 40 us; skew 500 ppm
I20260925 10:32:13.017063 2112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:13 dist-test-slave-dxdd krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:13.020612 2112 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:32:13.022061 2112 webserver.cc:460] Webserver started at http://127.0.97.62:41147/ using document root <none> and password file <none>
I20260925 10:32:13.022783 2112 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:13.022903 2112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:13.023205 2112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:13.025983 2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "cdcc438de4d74eeaade5e4b60a337d7e"
format_stamp: "Formatted at 2026-09-25 10:32:13 on dist-test-slave-dxdd"
server_key: "69b9a1ea5e2819b7a470a093aa22c1b5"
server_key_iv: "36c22ea669ddc06cbdec9118de9a13d3"
server_key_version: "encryptionkey@0"
I20260925 10:32:13.026789 2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "cdcc438de4d74eeaade5e4b60a337d7e"
format_stamp: "Formatted at 2026-09-25 10:32:13 on dist-test-slave-dxdd"
server_key: "69b9a1ea5e2819b7a470a093aa22c1b5"
server_key_iv: "36c22ea669ddc06cbdec9118de9a13d3"
server_key_version: "encryptionkey@0"
I20260925 10:32:13.031546 2112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 10:32:13.034806 2127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:13.036068 2112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 10:32:13.036238 2112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "cdcc438de4d74eeaade5e4b60a337d7e"
format_stamp: "Formatted at 2026-09-25 10:32:13 on dist-test-slave-dxdd"
server_key: "69b9a1ea5e2819b7a470a093aa22c1b5"
server_key_iv: "36c22ea669ddc06cbdec9118de9a13d3"
server_key_version: "encryptionkey@0"
I20260925 10:32:13.036438 2112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:13.058174 2112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:13.061786 2112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:13.062040 2112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:13.082756 2112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34799
I20260925 10:32:13.082775 2188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34799 every 8 connection(s)
I20260925 10:32:13.084007 2112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:32:13.087460 2189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:13.089679 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 2112
I20260925 10:32:13.089890 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:32:13.090543 388 external_mini_cluster.cc:1428] Setting key 43938bc07402339d8e5a8ab98008eb9f
I20260925 10:32:13.098788 2189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e: Bootstrap starting.
I20260925 10:32:13.102401 2189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:13.103688 2189 log.cc:824] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:32:13 dist-test-slave-dxdd krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:13.106861 2189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e: No bootstrap required, opened a new log
I20260925 10:32:13.113235 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:13.092374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.0.97.62:34799)
0925 10:32:13.093089 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:13.093099 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:13.093136 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:32:13.094751 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:13.094770 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:13.095136 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:13.095451 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:13.097116 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:13.098522 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:13.100254 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:13.100894 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:13.101032 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:13.106092 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:13.106099 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:13.106111 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:13.106146 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:13.109216 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:13.110007 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:13.110013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:13.110018 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:13.110160 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:13.110590 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:13.110593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:13.110595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:13.111067 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:13.111270 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:13.111804 (+ 534us) server_negotiation.cc:299] Negotiation successful
0925 10:32:13.112213 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":336,"threads_started":1}
I20260925 10:32:13.115928 2189 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcc438de4d74eeaade5e4b60a337d7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } }
I20260925 10:32:13.116282 2189 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:13.116389 2189 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdcc438de4d74eeaade5e4b60a337d7e, State: Initialized, Role: FOLLOWER
I20260925 10:32:13.116890 2189 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcc438de4d74eeaade5e4b60a337d7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } }
I20260925 10:32:13.117136 2189 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:32:13.117274 2189 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:32:13.117440 2189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:13.119529 2189 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcc438de4d74eeaade5e4b60a337d7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } }
I20260925 10:32:13.120083 2189 leader_election.cc:302] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdcc438de4d74eeaade5e4b60a337d7e; no voters:
I20260925 10:32:13.120661 2189 leader_election.cc:288] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:32:13.120752 2194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:13.121454 2194 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [term 1 LEADER]: Becoming Leader. State: Replica: cdcc438de4d74eeaade5e4b60a337d7e, State: Running, Role: LEADER
I20260925 10:32:13.121982 2194 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcc438de4d74eeaade5e4b60a337d7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } }
I20260925 10:32:13.122691 2189 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:13.124410 2195 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdcc438de4d74eeaade5e4b60a337d7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcc438de4d74eeaade5e4b60a337d7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } } }
I20260925 10:32:13.124768 2195 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:13.126269 2196 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdcc438de4d74eeaade5e4b60a337d7e. Latest consensus state: current_term: 1 leader_uuid: "cdcc438de4d74eeaade5e4b60a337d7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcc438de4d74eeaade5e4b60a337d7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } } }
I20260925 10:32:13.126598 2196 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:13.126750 2200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:13.130769 2200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:13.140290 2200 catalog_manager.cc:1347] Generated new cluster ID: 89026d29f5df4d47850b64701e2c1452
I20260925 10:32:13.140381 2200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:13.159646 2200 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:32:13.160915 2200 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:13.168818 2200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e: Generated new TSK 0
I20260925 10:32:13.169584 2200 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:32:13.219367 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34799
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:13.529707 2215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:13.529965 2215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:13.530131 2215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:13.537472 2215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:13.537595 2215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:13.537719 2215 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:32:13.537823 2215 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:32:13.537946 2215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:32:13.547744 2215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:13.548913 2215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:13.550179 2215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:13.562774 2220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:13.562814 2221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:13.562837 2223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:13.563897 2215 server_base.cc:1034] running on GCE node
I20260925 10:32:13.564448 2215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:13.565346 2215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:13.566565 2215 hybrid_clock.cc:648] HybridClock initialized: now 1790332333566513 us; error 47 us; skew 500 ppm
I20260925 10:32:13.566854 2215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:13 dist-test-slave-dxdd krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:13.569710 2215 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:32:13.570987 2215 webserver.cc:460] Webserver started at http://127.0.97.1:35337/ using document root <none> and password file <none>
I20260925 10:32:13.571556 2215 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:13.571657 2215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:13.571883 2215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:13.574731 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "ce4182079abb4ae0bfd02ce5d1e141df"
format_stamp: "Formatted at 2026-09-25 10:32:13 on dist-test-slave-dxdd"
server_key: "f8fd8c7591e5d37a3dd9f098a684adec"
server_key_iv: "94f221449be02569595984bbfe8d0695"
server_key_version: "encryptionkey@0"
I20260925 10:32:13.575408 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ce4182079abb4ae0bfd02ce5d1e141df"
format_stamp: "Formatted at 2026-09-25 10:32:13 on dist-test-slave-dxdd"
server_key: "f8fd8c7591e5d37a3dd9f098a684adec"
server_key_iv: "94f221449be02569595984bbfe8d0695"
server_key_version: "encryptionkey@0"
I20260925 10:32:13.579991 2215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:32:13.583118 2230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:13.584367 2215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:32:13.584520 2215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "ce4182079abb4ae0bfd02ce5d1e141df"
format_stamp: "Formatted at 2026-09-25 10:32:13 on dist-test-slave-dxdd"
server_key: "f8fd8c7591e5d37a3dd9f098a684adec"
server_key_iv: "94f221449be02569595984bbfe8d0695"
server_key_version: "encryptionkey@0"
I20260925 10:32:13.584743 2215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:13.596638 2215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:13.600492 2215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:13.600766 2215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:13.603209 2215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:13.603331 2215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:13.603518 2215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:13.603643 2215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:32:13 dist-test-slave-dxdd krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:13.626643 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:13.603531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59707 (local address 127.0.97.62:34799)
0925 10:32:13.603722 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:13.603728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:13.605175 (+ 1447us) server_negotiation.cc:407] Connection header received
0925 10:32:13.608007 (+ 2832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:13.608013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:13.608096 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:13.608217 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:13.610574 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:13.611532 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:13.614069 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:13.614497 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:13.614630 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:13.619119 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:13.619129 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:13.619137 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:13.619190 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:13.622998 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:13.623950 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:13.623956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:13.623960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:13.624115 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:13.624666 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:13.624671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:13.624674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:13.625043 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:13.625191 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:13.626013 (+ 822us) server_negotiation.cc:299] Negotiation successful
0925 10:32:13.626229 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:32:13.628080 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:13.604166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34799 (local address 127.0.97.1:59707)
0925 10:32:13.604994 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:13.605056 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:13.607703 (+ 2647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:13.608463 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:13.608483 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:13.609334 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:13.610381 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:13.610409 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:13.611683 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:13.611689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:13.613914 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:13.613937 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:13.614810 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:13.614817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:13.614985 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:13.615763 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:13.615786 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:13.618847 (+ 3061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:13.623152 (+ 4305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:13.623163 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:13.623189 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:13.623766 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:13.624253 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:13.624260 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:13.624265 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:13.624534 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:13.625157 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:13.625164 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:13.625629 (+ 465us) client_negotiation.cc:769] Sending connection context
0925 10:32:13.625934 (+ 305us) client_negotiation.cc:240] Negotiation successful
0925 10:32:13.626460 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":335,"threads_started":1}
I20260925 10:32:13.649482 2215 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41351
I20260925 10:32:13.649570 2349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41351 every 8 connection(s)
I20260925 10:32:13.650941 2215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:32:13.653097 2350 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34799
I20260925 10:32:13.653295 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:13.636234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38099 (local address 127.0.97.62:34799)
0925 10:32:13.636417 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:13.636424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:13.637315 (+ 891us) server_negotiation.cc:407] Connection header received
0925 10:32:13.637429 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:13.637437 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:13.637526 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:13.637625 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:13.639566 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:13.640799 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:13.643238 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:13.643976 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:13.644123 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:13.647048 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:13.647056 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:13.647064 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:13.647110 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:13.650291 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:13.651135 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:13.651138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:13.651140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:13.651255 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:13.651728 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:13.651737 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:13.651743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:13.652052 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:13.652213 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:13.652776 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 10:32:13.652956 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:32:13.653328 2350 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:13.653366 2311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:13.636288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34799 (local address 127.0.97.1:38099)
0925 10:32:13.637187 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:13.637207 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:13.637334 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:13.637843 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:13.637854 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:13.638773 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:13.639313 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:13.639334 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:13.641070 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:13.641080 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:13.643024 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:13.643039 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:13.644212 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:13.644221 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:13.644438 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:13.645362 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:13.645394 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:13.646866 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:13.650454 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:13.650463 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:13.650468 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:13.650975 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:13.651367 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:13.651377 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:13.651383 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:13.651552 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:13.652311 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:13.652335 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:13.652514 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 10:32:13.652646 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 10:32:13.652933 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":613,"threads_started":1}
I20260925 10:32:13.654162 2350 heartbeater.cc:511] Master 127.0.97.62:34799 requested a full tablet report, sending...
I20260925 10:32:13.655256 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 2215
I20260925 10:32:13.655467 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:32:13.656122 388 external_mini_cluster.cc:1428] Setting key d2d7a65fbbcff95017f3dab28cae87c6
I20260925 10:32:13.656128 2144 ts_manager.cc:194] Registered new tserver with Master: ce4182079abb4ae0bfd02ce5d1e141df (127.0.97.1:41351)
I20260925 10:32:13.657799 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59707
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:32:13.693163 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34799
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:13.994652 2354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:13.994910 2354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:13.995008 2354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:14.002332 2354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:14.002439 2354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:14.002521 2354 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:32:14.002573 2354 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:32:14.002646 2354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:32:14.011915 2354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:14.013093 2354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:14.014344 2354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:14.026993 2359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:14.026955 2360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:14.027036 2362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:14.027827 2354 server_base.cc:1034] running on GCE node
I20260925 10:32:14.028421 2354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:14.029291 2354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:14.030511 2354 hybrid_clock.cc:648] HybridClock initialized: now 1790332334030448 us; error 58 us; skew 500 ppm
I20260925 10:32:14.030810 2354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:14 dist-test-slave-dxdd krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:14.037976 2354 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:32:14.039459 2354 webserver.cc:460] Webserver started at http://127.0.97.2:32943/ using document root <none> and password file <none>
I20260925 10:32:14.040148 2354 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:14.040272 2354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:14.040572 2354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:14.043396 2354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "b0729ceedca9436d8691873a05be12ce"
format_stamp: "Formatted at 2026-09-25 10:32:14 on dist-test-slave-dxdd"
server_key: "0e2608799dc25211584c86f96a610095"
server_key_iv: "2aaaf90dca75bb8dbe91649c82a7133d"
server_key_version: "encryptionkey@0"
I20260925 10:32:14.044133 2354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b0729ceedca9436d8691873a05be12ce"
format_stamp: "Formatted at 2026-09-25 10:32:14 on dist-test-slave-dxdd"
server_key: "0e2608799dc25211584c86f96a610095"
server_key_iv: "2aaaf90dca75bb8dbe91649c82a7133d"
server_key_version: "encryptionkey@0"
I20260925 10:32:14.049140 2354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 10:32:14.052390 2369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:14.053550 2354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:32:14.053714 2354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "b0729ceedca9436d8691873a05be12ce"
format_stamp: "Formatted at 2026-09-25 10:32:14 on dist-test-slave-dxdd"
server_key: "0e2608799dc25211584c86f96a610095"
server_key_iv: "2aaaf90dca75bb8dbe91649c82a7133d"
server_key_version: "encryptionkey@0"
I20260925 10:32:14.053879 2354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:14.067302 2354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:14.070966 2354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:14.071229 2354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:14.073549 2354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:14.073662 2354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:14.073777 2354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:14.073868 2354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:32:14 dist-test-slave-dxdd krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:14.098796 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.074274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45461 (local address 127.0.97.62:34799)
0925 10:32:14.074493 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:14.074500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:14.075845 (+ 1345us) server_negotiation.cc:407] Connection header received
0925 10:32:14.078032 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:14.078037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:14.078119 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:14.078240 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:14.080361 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.081639 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.084541 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.085230 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.085342 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.090203 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:14.090213 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:14.090220 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:14.090281 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:14.094597 (+ 4316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:14.095762 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:14.095770 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:14.095776 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:14.095948 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:14.096484 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:14.096492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:14.096497 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:14.096920 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:14.098059 (+ 1139us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:14.098222 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 10:32:14.098482 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 10:32:14.100492 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.074883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34799 (local address 127.0.97.2:45461)
0925 10:32:14.075697 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:14.075731 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:14.077767 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:14.078527 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:14.078541 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:14.079353 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:14.080145 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.080190 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.081819 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.081824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.084340 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.084354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.085659 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.085667 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.085938 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.086751 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:14.086780 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:14.089900 (+ 3120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:14.094779 (+ 4879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:14.094792 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:14.094812 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:14.095552 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:14.096114 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:14.096119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:14.096123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:14.096346 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:14.097101 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:14.097114 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:14.097649 (+ 535us) client_negotiation.cc:769] Sending connection context
0925 10:32:14.097997 (+ 348us) client_negotiation.cc:240] Negotiation successful
0925 10:32:14.098699 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":365,"threads_started":1}
I20260925 10:32:14.122080 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.105953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33365 (local address 127.0.97.62:34799)
0925 10:32:14.106159 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:14.106164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:14.107731 (+ 1567us) server_negotiation.cc:407] Connection header received
0925 10:32:14.107848 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:14.107856 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:14.107963 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:14.108093 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:14.108954 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.110512 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.112549 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.113330 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.113479 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.115808 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:14.115813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:14.115816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:14.115844 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:14.119153 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:14.120052 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:14.120059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:14.120063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:14.120177 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:14.120539 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:14.120546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:14.120552 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:14.120850 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:14.120997 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:14.121520 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 10:32:14.121739 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:32:14.122246 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.106485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34799 (local address 127.0.97.2:33365)
0925 10:32:14.107621 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:14.107646 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:14.107751 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:14.108145 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:14.108150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:14.108542 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:14.108819 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.108827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.110676 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.110683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.112340 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.112357 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.113540 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.113546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.113692 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.114470 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:14.114486 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:14.115601 (+ 1115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:14.119302 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:14.119307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:14.119311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:14.119911 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:14.120290 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:14.120295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:14.120298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:14.120431 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:14.120971 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:14.120976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:14.121216 (+ 240us) client_negotiation.cc:769] Sending connection context
0925 10:32:14.121437 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 10:32:14.121752 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"thread_start_us":729,"threads_started":1}
I20260925 10:32:14.129658 2354 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44639
I20260925 10:32:14.129693 2488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44639 every 8 connection(s)
I20260925 10:32:14.130905 2354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:32:14.132807 2489 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34799
I20260925 10:32:14.132966 2489 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:14.133476 2489 heartbeater.cc:511] Master 127.0.97.62:34799 requested a full tablet report, sending...
I20260925 10:32:14.134603 2144 ts_manager.cc:194] Registered new tserver with Master: b0729ceedca9436d8691873a05be12ce (127.0.97.2:44639)
I20260925 10:32:14.135531 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45461
I20260925 10:32:14.138845 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 2354
I20260925 10:32:14.139012 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:32:14.139438 388 external_mini_cluster.cc:1428] Setting key 240c2253b7e8783b7266acd3404b2abf
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:32:14.176967 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34799
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:14.481621 2493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:14.481878 2493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:14.481976 2493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:14.489023 2493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:14.489125 2493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:14.489213 2493 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:32:14.489262 2493 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:32:14.489323 2493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:32:14.498720 2493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:14.499842 2493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:14.501173 2493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:14.513516 2499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:14.513481 2501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:14.514154 2498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:14.515054 2493 server_base.cc:1034] running on GCE node
I20260925 10:32:14.515671 2493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:14.516525 2493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:14.517745 2493 hybrid_clock.cc:648] HybridClock initialized: now 1790332334517700 us; error 57 us; skew 500 ppm
I20260925 10:32:14.518044 2493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:14 dist-test-slave-dxdd krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:14.521097 2493 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:32:14.522532 2493 webserver.cc:460] Webserver started at http://127.0.97.3:34613/ using document root <none> and password file <none>
I20260925 10:32:14.523173 2493 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:14.523299 2493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:14.523639 2493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:14.526432 2493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "98b2ae42644544c3ba0d44d87e02e76a"
format_stamp: "Formatted at 2026-09-25 10:32:14 on dist-test-slave-dxdd"
server_key: "bf6023d03ed374fa8c3dc5ff273bb275"
server_key_iv: "882a19453d5e28700a36953305a9886c"
server_key_version: "encryptionkey@0"
I20260925 10:32:14.527158 2493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "98b2ae42644544c3ba0d44d87e02e76a"
format_stamp: "Formatted at 2026-09-25 10:32:14 on dist-test-slave-dxdd"
server_key: "bf6023d03ed374fa8c3dc5ff273bb275"
server_key_iv: "882a19453d5e28700a36953305a9886c"
server_key_version: "encryptionkey@0"
I20260925 10:32:14.532024 2493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:32:14.535365 2508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:14.536576 2493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 10:32:14.536749 2493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "98b2ae42644544c3ba0d44d87e02e76a"
format_stamp: "Formatted at 2026-09-25 10:32:14 on dist-test-slave-dxdd"
server_key: "bf6023d03ed374fa8c3dc5ff273bb275"
server_key_iv: "882a19453d5e28700a36953305a9886c"
server_key_version: "encryptionkey@0"
I20260925 10:32:14.536926 2493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:14.561302 2493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:14.565491 2493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:14.565774 2493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:14.568334 2493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:14.568418 2493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:14.568547 2493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:14.568619 2493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:32:14 dist-test-slave-dxdd krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:14.592752 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.569180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37413 (local address 127.0.97.62:34799)
0925 10:32:14.569346 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:14.569351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:14.570957 (+ 1606us) server_negotiation.cc:407] Connection header received
0925 10:32:14.573057 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:14.573062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:14.573148 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:14.573253 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:14.575948 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.576913 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.580059 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.580596 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.580735 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.585520 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:14.585524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:14.585528 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:14.585586 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:14.589042 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:14.590152 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:14.590161 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:14.590166 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:14.590451 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:14.591021 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:14.591024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:14.591026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:14.591289 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:14.591398 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:14.592298 (+ 900us) server_negotiation.cc:299] Negotiation successful
0925 10:32:14.592516 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:32:14.594590 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.569840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34799 (local address 127.0.97.3:37413)
0925 10:32:14.570775 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:14.570840 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:14.572773 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:14.573649 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:14.573664 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:14.574661 (+ 997us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:14.575736 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.575770 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.577104 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.577117 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.579887 (+ 2770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.579903 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.580878 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.580890 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.581248 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.582177 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:14.582224 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:14.585231 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:14.589182 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:14.589194 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:14.589223 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:14.589950 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:14.590580 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:14.590588 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:14.590594 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:14.590898 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:14.591417 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:14.591428 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:14.591906 (+ 478us) client_negotiation.cc:769] Sending connection context
0925 10:32:14.592246 (+ 340us) client_negotiation.cc:240] Negotiation successful
0925 10:32:14.592871 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":410,"threads_started":1}
I20260925 10:32:14.622611 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.607222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34799 (local address 127.0.97.3:42381)
0925 10:32:14.607865 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:14.607901 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:14.608049 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:14.608359 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:14.608364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:14.608825 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:14.609080 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.609088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.610217 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.610224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.611850 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.611864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.612767 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.612776 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.613021 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.613779 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:14.613806 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:14.614941 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:14.618996 (+ 4055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:14.619004 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:14.619009 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:14.619581 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:14.620484 (+ 903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:14.620489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:14.620493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:14.620645 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:14.621396 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:14.621401 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:14.621621 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 10:32:14.621795 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 10:32:14.622119 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":359,"threads_started":1}
I20260925 10:32:14.623806 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.607132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42381 (local address 127.0.97.62:34799)
0925 10:32:14.607337 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:14.607343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:14.607983 (+ 640us) server_negotiation.cc:407] Connection header received
0925 10:32:14.608163 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:14.608168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:14.608238 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:14.608342 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:14.609215 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.610099 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.611993 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.612514 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.612688 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.615133 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:14.615140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:14.615147 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:14.615189 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:14.618847 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:14.619764 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:14.619772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:14.619778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:14.620063 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:14.620781 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:14.620785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:14.620787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:14.621180 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:14.621335 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:14.623283 (+ 1948us) server_negotiation.cc:299] Negotiation successful
0925 10:32:14.623496 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:32:14.630956 2493 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33975
I20260925 10:32:14.630986 2627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33975 every 8 connection(s)
I20260925 10:32:14.632239 2493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:32:14.633546 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 2493
I20260925 10:32:14.633703 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:32:14.634133 388 external_mini_cluster.cc:1428] Setting key 954a09fa14f95ed0a617efd50d11985f
I20260925 10:32:14.634166 2628 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34799
I20260925 10:32:14.634739 2628 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:14.635486 2628 heartbeater.cc:511] Master 127.0.97.62:34799 requested a full tablet report, sending...
I20260925 10:32:14.636960 2143 ts_manager.cc:194] Registered new tserver with Master: 98b2ae42644544c3ba0d44d87e02e76a (127.0.97.3:33975)
I20260925 10:32:14.637785 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37413
I20260925 10:32:14.648856 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:32:14.660336 2350 heartbeater.cc:503] Master 127.0.97.62:34799 was elected leader, sending a full tablet report...
Sep 25 10:32:14 dist-test-slave-dxdd krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:32:14 dist-test-slave-dxdd krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:14.693151 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.679873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.0.97.62:34799)
0925 10:32:14.680092 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:14.680099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:14.680380 (+ 281us) server_negotiation.cc:407] Connection header received
0925 10:32:14.680574 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:14.680581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:14.680654 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:14.680769 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:14.682042 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.682809 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.684039 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.684443 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.684568 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.688430 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:14.688435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:14.688438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:14.688469 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:14.690804 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:14.691381 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:14.691388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:14.691396 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:14.691498 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:14.691846 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:14.691852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:14.691855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:14.692098 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:14.692337 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:14.692644 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 10:32:14.692860 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:32:14.696863 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:32:14.703495 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:32:14.726372 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.714073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33975 (local address 127.0.0.1:54714)
0925 10:32:14.714958 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:14.714995 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:14.715158 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:14.716635 (+ 1477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:14.716642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:14.716671 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:14.717125 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.717141 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.719802 (+ 2661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.719811 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.722759 (+ 2948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.722778 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.725050 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.725059 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.725350 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.725427 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 10:32:14.725552 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 10:32:14.725796 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":395,"threads_started":1}
I20260925 10:32:14.728317 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.714276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.0.97.3:33975)
0925 10:32:14.715898 (+ 1622us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:14.715909 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:14.715953 (+ 44us) server_negotiation.cc:407] Connection header received
0925 10:32:14.716058 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:14.716066 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:14.716384 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:14.717355 (+ 971us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:14.717684 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.719627 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.723005 (+ 3378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.724783 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.727250 (+ 2467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.727556 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:14.727644 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 10:32:14.727862 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1333,"thread_start_us":2265,"threads_started":1}
I20260925 10:32:14.732539 2558 tablet_service.cc:1467] Processing CreateTablet for tablet d67c4bafadb54da5aad75beb2f1dbdf7 (DEFAULT_TABLE table=test-table [id=eb34fd366e1b4296a9b0afd7de7d87be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:14.733927 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.714510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44639 (local address 127.0.0.1:41774)
0925 10:32:14.723837 (+ 9327us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:14.723861 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:14.723997 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:14.725780 (+ 1783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:14.725787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:14.725822 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:14.726328 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.726342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.728335 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.728343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.731456 (+ 3113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.731467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.733257 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.733264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.733389 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.733411 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 10:32:14.733502 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 10:32:14.733620 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9139,"thread_start_us":16661,"threads_started":1}
I20260925 10:32:14.733985 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67c4bafadb54da5aad75beb2f1dbdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:14.734236 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.715035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41774 (local address 127.0.97.2:44639)
0925 10:32:14.724787 (+ 9752us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:14.724799 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:14.724835 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:32:14.724959 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:14.724970 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:14.725279 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:14.725511 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:14.726545 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.728117 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.731649 (+ 3532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.732847 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.732982 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.733113 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:14.733570 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 10:32:14.733731 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9556,"thread_start_us":9391,"threads_started":1}
I20260925 10:32:14.739045 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.715381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41351 (local address 127.0.0.1:44522)
0925 10:32:14.726615 (+ 11234us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:14.726631 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:14.726758 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:14.732035 (+ 5277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:14.732039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:14.732062 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:14.732293 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.732303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.734185 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.734190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.735881 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.735894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.738326 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.738331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.738501 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.738535 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 10:32:14.738616 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 10:32:14.738760 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11152,"thread_start_us":8733,"threads_started":1}
I20260925 10:32:14.738860 2419 tablet_service.cc:1467] Processing CreateTablet for tablet d67c4bafadb54da5aad75beb2f1dbdf7 (DEFAULT_TABLE table=test-table [id=eb34fd366e1b4296a9b0afd7de7d87be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:14.740000 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.715618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44522 (local address 127.0.97.1:41351)
0925 10:32:14.731451 (+ 15833us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:14.731459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:14.731484 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:32:14.731571 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:14.731577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:14.731837 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:14.732479 (+ 642us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:14.732666 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.734020 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.736792 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.738143 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.739296 (+ 1153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.739492 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:14.739554 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 10:32:14.739721 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15705,"thread_start_us":15424,"threads_started":1}
I20260925 10:32:14.740128 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67c4bafadb54da5aad75beb2f1dbdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:14.743438 2280 tablet_service.cc:1467] Processing CreateTablet for tablet d67c4bafadb54da5aad75beb2f1dbdf7 (DEFAULT_TABLE table=test-table [id=eb34fd366e1b4296a9b0afd7de7d87be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:14.745025 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67c4bafadb54da5aad75beb2f1dbdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:14.746850 2647 tablet_bootstrap.cc:492] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a: Bootstrap starting.
I20260925 10:32:14.749745 2647 tablet_bootstrap.cc:654] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:14.750780 2647 log.cc:824] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:14.753417 2647 tablet_bootstrap.cc:492] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a: No bootstrap required, opened a new log
I20260925 10:32:14.753670 2647 ts_tablet_manager.cc:1397] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260925 10:32:14.754588 2649 tablet_bootstrap.cc:492] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce: Bootstrap starting.
I20260925 10:32:14.757822 2649 tablet_bootstrap.cc:654] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:14.758102 2650 tablet_bootstrap.cc:492] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df: Bootstrap starting.
I20260925 10:32:14.759120 2649 log.cc:824] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:14.761449 2650 tablet_bootstrap.cc:654] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:14.761996 2649 tablet_bootstrap.cc:492] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce: No bootstrap required, opened a new log
I20260925 10:32:14.762272 2649 ts_tablet_manager.cc:1397] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260925 10:32:14.762545 2650 log.cc:824] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:14.763501 2647 raft_consensus.cc:348] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } }
I20260925 10:32:14.763931 2647 raft_consensus.cc:374] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:14.764029 2647 raft_consensus.cc:729] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98b2ae42644544c3ba0d44d87e02e76a, State: Initialized, Role: FOLLOWER
I20260925 10:32:14.764674 2647 consensus_queue.cc:260] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } }
I20260925 10:32:14.765725 2628 heartbeater.cc:503] Master 127.0.97.62:34799 was elected leader, sending a full tablet report...
I20260925 10:32:14.766367 2647 ts_tablet_manager.cc:1428] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260925 10:32:14.766409 2650 tablet_bootstrap.cc:492] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df: No bootstrap required, opened a new log
I20260925 10:32:14.766762 2650 ts_tablet_manager.cc:1397] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260925 10:32:14.770394 2649 raft_consensus.cc:348] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } }
I20260925 10:32:14.770723 2649 raft_consensus.cc:374] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:14.770843 2649 raft_consensus.cc:729] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0729ceedca9436d8691873a05be12ce, State: Initialized, Role: FOLLOWER
I20260925 10:32:14.771447 2649 consensus_queue.cc:260] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } }
I20260925 10:32:14.772490 2489 heartbeater.cc:503] Master 127.0.97.62:34799 was elected leader, sending a full tablet report...
I20260925 10:32:14.772962 2649 ts_tablet_manager.cc:1428] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260925 10:32:14.776151 2650 raft_consensus.cc:348] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } }
I20260925 10:32:14.776540 2650 raft_consensus.cc:374] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:14.776645 2650 raft_consensus.cc:729] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4182079abb4ae0bfd02ce5d1e141df, State: Initialized, Role: FOLLOWER
I20260925 10:32:14.777240 2650 consensus_queue.cc:260] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } }
I20260925 10:32:14.778697 2650 ts_tablet_manager.cc:1428] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260925 10:32:14.789551 2654 raft_consensus.cc:482] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:32:14.789762 2654 raft_consensus.cc:504] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } }
I20260925 10:32:14.791105 2654 leader_election.cc:288] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98b2ae42644544c3ba0d44d87e02e76a (127.0.97.3:33975), ce4182079abb4ae0bfd02ce5d1e141df (127.0.97.1:41351)
I20260925 10:32:14.798602 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.791545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34935 (local address 127.0.97.3:33975)
0925 10:32:14.791771 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:14.791777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:14.792081 (+ 304us) server_negotiation.cc:407] Connection header received
0925 10:32:14.792297 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:14.792302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:14.792370 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:14.792438 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:14.793210 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.794142 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.796597 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.797779 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.797903 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.798006 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:14.798220 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 10:32:14.798383 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:32:14.798700 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.791384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33975 (local address 127.0.97.2:34935)
0925 10:32:14.791976 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:14.791995 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:14.792127 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:14.792656 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:14.792660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:14.792685 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:14.793078 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.793091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.794263 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.794266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.796456 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.796476 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.797954 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.797958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.798063 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.798086 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:32:14.798158 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 10:32:14.798236 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":51,"thread_start_us":277,"threads_started":1}
I20260925 10:32:14.799207 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.792140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42489 (local address 127.0.97.1:41351)
0925 10:32:14.792371 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:14.792380 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:14.792875 (+ 495us) server_negotiation.cc:407] Connection header received
0925 10:32:14.793058 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:14.793065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:14.793151 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:14.793242 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:14.793740 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.794641 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.796950 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.798221 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.798363 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.798452 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:14.798830 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 10:32:14.798979 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:32:14.799702 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.791992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41351 (local address 127.0.97.2:42489)
0925 10:32:14.792764 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:14.792784 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:14.792901 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:14.793265 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:14.793269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:14.793292 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:14.793585 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.793593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.794777 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.794782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.796822 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.796832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.798449 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.798455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.798625 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.798671 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 10:32:14.799367 (+ 696us) client_negotiation.cc:240] Negotiation successful
0925 10:32:14.799467 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":22,"thread_start_us":538,"threads_started":1}
I20260925 10:32:14.799674 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67c4bafadb54da5aad75beb2f1dbdf7" candidate_uuid: "b0729ceedca9436d8691873a05be12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98b2ae42644544c3ba0d44d87e02e76a" is_pre_election: true
I20260925 10:32:14.800231 2578 raft_consensus.cc:2454] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0729ceedca9436d8691873a05be12ce in term 0.
I20260925 10:32:14.800587 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67c4bafadb54da5aad75beb2f1dbdf7" candidate_uuid: "b0729ceedca9436d8691873a05be12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" is_pre_election: true
I20260925 10:32:14.800858 2373 leader_election.cc:302] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98b2ae42644544c3ba0d44d87e02e76a, b0729ceedca9436d8691873a05be12ce; no voters:
I20260925 10:32:14.801041 2304 raft_consensus.cc:2454] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0729ceedca9436d8691873a05be12ce in term 0.
I20260925 10:32:14.801273 2654 raft_consensus.cc:2781] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:32:14.801434 2654 raft_consensus.cc:482] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:32:14.801553 2654 raft_consensus.cc:3037] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:14.803757 2654 raft_consensus.cc:504] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } }
I20260925 10:32:14.804255 2654 leader_election.cc:288] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [CANDIDATE]: Term 1 election: Requested vote from peers 98b2ae42644544c3ba0d44d87e02e76a (127.0.97.3:33975), ce4182079abb4ae0bfd02ce5d1e141df (127.0.97.1:41351)
I20260925 10:32:14.804675 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67c4bafadb54da5aad75beb2f1dbdf7" candidate_uuid: "b0729ceedca9436d8691873a05be12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98b2ae42644544c3ba0d44d87e02e76a"
I20260925 10:32:14.804719 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67c4bafadb54da5aad75beb2f1dbdf7" candidate_uuid: "b0729ceedca9436d8691873a05be12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4182079abb4ae0bfd02ce5d1e141df"
I20260925 10:32:14.804893 2304 raft_consensus.cc:3037] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:14.804893 2578 raft_consensus.cc:3037] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:14.807973 2578 raft_consensus.cc:2454] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0729ceedca9436d8691873a05be12ce in term 1.
I20260925 10:32:14.807973 2304 raft_consensus.cc:2454] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0729ceedca9436d8691873a05be12ce in term 1.
I20260925 10:32:14.808324 2373 leader_election.cc:302] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98b2ae42644544c3ba0d44d87e02e76a, b0729ceedca9436d8691873a05be12ce; no voters:
I20260925 10:32:14.808565 2654 raft_consensus.cc:2781] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:14.808969 2654 raft_consensus.cc:686] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [term 1 LEADER]: Becoming Leader. State: Replica: b0729ceedca9436d8691873a05be12ce, State: Running, Role: LEADER
I20260925 10:32:14.809368 2654 consensus_queue.cc:237] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } }
I20260925 10:32:14.813787 2143 catalog_manager.cc:5495] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce reported cstate change: term changed from 0 to 1, leader changed from <none> to b0729ceedca9436d8691873a05be12ce (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b0729ceedca9436d8691873a05be12ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } health_report { overall_health: HEALTHY } } }
Sep 25 10:32:14 dist-test-slave-dxdd krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:14.880417 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.868431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34799 (local address 127.0.0.1:51948)
0925 10:32:14.868971 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:14.868986 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:14.869103 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:14.869378 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:14.869381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:14.870016 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:14.870269 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.870277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.871215 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.871218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.872160 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:14.872170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.872898 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.872902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:14.873022 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.873614 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:14.873630 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:14.875755 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:14.878361 (+ 2606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:14.878377 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:14.878382 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:14.878761 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:14.879057 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:14.879060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:14.879062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:14.879194 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:14.879626 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:14.879631 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:14.879766 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 10:32:14.879897 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 10:32:14.880080 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":297,"threads_started":1}
I20260925 10:32:14.880523 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:14.868431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.0.97.62:34799)
0925 10:32:14.868583 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:14.868588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:14.869072 (+ 484us) server_negotiation.cc:407] Connection header received
0925 10:32:14.869200 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:14.869203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:14.869260 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:14.869339 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:14.870414 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.871103 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.872312 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:14.872741 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:14.872847 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:14.875923 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:14.875928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:14.875931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:14.875956 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:14.878198 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:14.878853 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:14.878857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:14.878859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:14.878970 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:14.879284 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:14.879287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:14.879289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:14.879529 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:14.879627 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:14.879985 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 10:32:14.880201 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":56}
W20260925 10:32:14.882789 2490 tablet.cc:2369] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:32:14.883921 2629 tablet.cc:2369] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:32:14.885388 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51948:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:32:14.886245 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:32:14.892202 2419 tablet_service.cc:1467] Processing CreateTablet for tablet c65f987f616b413e8453428bba31234b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a02bb82f95d840d2b3eaa797d5b11e80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:32:14.892123 2280 tablet_service.cc:1467] Processing CreateTablet for tablet c65f987f616b413e8453428bba31234b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a02bb82f95d840d2b3eaa797d5b11e80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:32:14.892907 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65f987f616b413e8453428bba31234b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:14.892923 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65f987f616b413e8453428bba31234b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:14.897238 2558 tablet_service.cc:1467] Processing CreateTablet for tablet c65f987f616b413e8453428bba31234b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a02bb82f95d840d2b3eaa797d5b11e80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:32:14.897897 2649 tablet_bootstrap.cc:492] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce: Bootstrap starting.
I20260925 10:32:14.898166 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65f987f616b413e8453428bba31234b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:14.898192 2650 tablet_bootstrap.cc:492] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df: Bootstrap starting.
I20260925 10:32:14.899596 2649 tablet_bootstrap.cc:654] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:14.899701 2650 tablet_bootstrap.cc:654] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:14.901875 2649 tablet_bootstrap.cc:492] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce: No bootstrap required, opened a new log
I20260925 10:32:14.902020 2649 ts_tablet_manager.cc:1397] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 10:32:14.902069 2650 tablet_bootstrap.cc:492] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df: No bootstrap required, opened a new log
I20260925 10:32:14.902617 2650 ts_tablet_manager.cc:1397] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260925 10:32:14.902850 2649 raft_consensus.cc:348] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:14.903082 2649 raft_consensus.cc:374] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:14.903172 2649 raft_consensus.cc:729] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0729ceedca9436d8691873a05be12ce, State: Initialized, Role: FOLLOWER
W20260925 10:32:14.903373 2351 tablet.cc:2369] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:32:14.903398 2649 consensus_queue.cc:260] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:14.903884 2654 tablet_replica.cc:439] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } }
I20260925 10:32:14.904068 2649 ts_tablet_manager.cc:1428] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:32:14.904186 2654 tablet_replica.cc:442] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:32:14.904590 2650 raft_consensus.cc:348] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:14.904804 2650 raft_consensus.cc:374] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:14.904889 2650 raft_consensus.cc:729] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4182079abb4ae0bfd02ce5d1e141df, State: Initialized, Role: FOLLOWER
I20260925 10:32:14.905138 2647 tablet_bootstrap.cc:492] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a: Bootstrap starting.
I20260925 10:32:14.905162 2650 consensus_queue.cc:260] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:14.905560 2655 tablet_replica.cc:439] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } }
I20260925 10:32:14.905885 2655 tablet_replica.cc:442] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:32:14.906721 2647 tablet_bootstrap.cc:654] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:14.908895 2650 ts_tablet_manager.cc:1428] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260925 10:32:14.911481 2647 tablet_bootstrap.cc:492] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a: No bootstrap required, opened a new log
I20260925 10:32:14.911629 2647 ts_tablet_manager.cc:1397] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260925 10:32:14.912318 2647 raft_consensus.cc:348] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:14.912524 2647 raft_consensus.cc:374] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:14.912618 2647 raft_consensus.cc:729] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98b2ae42644544c3ba0d44d87e02e76a, State: Initialized, Role: FOLLOWER
I20260925 10:32:14.912847 2647 consensus_queue.cc:260] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:14.913434 2647 ts_tablet_manager.cc:1428] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 10:32:14.913259 2653 tablet_replica.cc:439] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } }
I20260925 10:32:14.913627 2653 tablet_replica.cc:442] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:32:15.271042 2653 raft_consensus.cc:482] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:32:15.271222 2653 raft_consensus.cc:504] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:15.272612 2653 leader_election.cc:288] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce4182079abb4ae0bfd02ce5d1e141df (127.0.97.1:41351), b0729ceedca9436d8691873a05be12ce (127.0.97.2:44639)
I20260925 10:32:15.275465 2654 raft_consensus.cc:482] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:32:15.275645 2654 raft_consensus.cc:504] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:15.276283 2654 leader_election.cc:288] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce4182079abb4ae0bfd02ce5d1e141df (127.0.97.1:41351), 98b2ae42644544c3ba0d44d87e02e76a (127.0.97.3:33975)
I20260925 10:32:15.276733 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65f987f616b413e8453428bba31234b" candidate_uuid: "b0729ceedca9436d8691873a05be12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98b2ae42644544c3ba0d44d87e02e76a" is_pre_election: true
I20260925 10:32:15.276719 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65f987f616b413e8453428bba31234b" candidate_uuid: "b0729ceedca9436d8691873a05be12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" is_pre_election: true
I20260925 10:32:15.276958 2578 raft_consensus.cc:2454] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0729ceedca9436d8691873a05be12ce in term 0.
I20260925 10:32:15.276958 2304 raft_consensus.cc:2454] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0729ceedca9436d8691873a05be12ce in term 0.
I20260925 10:32:15.277402 2373 leader_election.cc:302] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98b2ae42644544c3ba0d44d87e02e76a, b0729ceedca9436d8691873a05be12ce; no voters:
I20260925 10:32:15.277673 2654 raft_consensus.cc:2781] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:32:15.277779 2654 raft_consensus.cc:482] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:32:15.277896 2654 raft_consensus.cc:3037] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:15.279868 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:15.272939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44087 (local address 127.0.97.1:41351)
0925 10:32:15.273162 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:15.273171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:15.273360 (+ 189us) server_negotiation.cc:407] Connection header received
0925 10:32:15.273636 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:15.273645 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:15.273732 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:15.273779 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:15.274446 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.275129 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.277724 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.278951 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.279088 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:15.279200 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:15.279492 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 10:32:15.279616 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 10:32:15.280182 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:15.272785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41351 (local address 127.0.97.3:44087)
0925 10:32:15.273251 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:15.273273 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:15.273470 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:15.273941 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:15.273944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:15.273963 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:15.274241 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:15.274258 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.275254 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.275262 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:15.277562 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:15.277574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.279115 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.279123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:15.279334 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:15.279372 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 10:32:15.279468 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 10:32:15.279604 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":79,"spinlock_wait_cycles":16256,"thread_start_us":253,"threads_started":1}
I20260925 10:32:15.280354 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:15.272870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44639 (local address 127.0.97.3:34927)
0925 10:32:15.273522 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:15.273538 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:15.273620 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:15.274039 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:15.274044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:15.274086 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:15.274439 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:15.274453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.275435 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.275441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:15.277526 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:15.277540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.279362 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.279371 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:15.279566 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:15.279598 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:32:15.279686 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:32:15.279808 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":22,"spinlock_wait_cycles":6272,"thread_start_us":388,"threads_started":1}
I20260925 10:32:15.280500 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:15.273356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34927 (local address 127.0.97.2:44639)
0925 10:32:15.273788 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:15.273793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:15.273807 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:32:15.273861 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:15.273864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:15.273921 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:15.273999 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:15.274587 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.275277 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.277721 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.279137 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.279275 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:15.279396 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:15.280002 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 10:32:15.280151 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":260,"threads_started":1}
I20260925 10:32:15.281064 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65f987f616b413e8453428bba31234b" candidate_uuid: "98b2ae42644544c3ba0d44d87e02e76a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" is_pre_election: true
I20260925 10:32:15.281306 2304 raft_consensus.cc:2454] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98b2ae42644544c3ba0d44d87e02e76a in term 0.
I20260925 10:32:15.281306 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65f987f616b413e8453428bba31234b" candidate_uuid: "98b2ae42644544c3ba0d44d87e02e76a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0729ceedca9436d8691873a05be12ce" is_pre_election: true
I20260925 10:32:15.280725 2654 raft_consensus.cc:504] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:15.281752 2444 raft_consensus.cc:2379] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 98b2ae42644544c3ba0d44d87e02e76a in current term 1: Already voted for candidate b0729ceedca9436d8691873a05be12ce in this term.
I20260925 10:32:15.281793 2512 leader_election.cc:302] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98b2ae42644544c3ba0d44d87e02e76a, ce4182079abb4ae0bfd02ce5d1e141df; no voters:
I20260925 10:32:15.282218 2654 leader_election.cc:288] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [CANDIDATE]: Term 1 election: Requested vote from peers ce4182079abb4ae0bfd02ce5d1e141df (127.0.97.1:41351), 98b2ae42644544c3ba0d44d87e02e76a (127.0.97.3:33975)
I20260925 10:32:15.282382 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65f987f616b413e8453428bba31234b" candidate_uuid: "b0729ceedca9436d8691873a05be12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4182079abb4ae0bfd02ce5d1e141df"
I20260925 10:32:15.282454 2653 raft_consensus.cc:2781] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:32:15.282577 2304 raft_consensus.cc:3037] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:15.282578 2653 raft_consensus.cc:482] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:32:15.282554 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65f987f616b413e8453428bba31234b" candidate_uuid: "b0729ceedca9436d8691873a05be12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98b2ae42644544c3ba0d44d87e02e76a"
I20260925 10:32:15.282687 2653 raft_consensus.cc:3037] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:15.284502 2304 raft_consensus.cc:2454] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0729ceedca9436d8691873a05be12ce in term 1.
I20260925 10:32:15.284837 2373 leader_election.cc:302] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0729ceedca9436d8691873a05be12ce, ce4182079abb4ae0bfd02ce5d1e141df; no voters:
I20260925 10:32:15.285069 2654 raft_consensus.cc:2781] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:15.285019 2653 raft_consensus.cc:504] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:15.285432 2578 raft_consensus.cc:2379] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b0729ceedca9436d8691873a05be12ce in current term 1: Already voted for candidate 98b2ae42644544c3ba0d44d87e02e76a in this term.
I20260925 10:32:15.285611 2653 leader_election.cc:288] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [CANDIDATE]: Term 1 election: Requested vote from peers ce4182079abb4ae0bfd02ce5d1e141df (127.0.97.1:41351), b0729ceedca9436d8691873a05be12ce (127.0.97.2:44639)
I20260925 10:32:15.285633 2654 raft_consensus.cc:686] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [term 1 LEADER]: Becoming Leader. State: Replica: b0729ceedca9436d8691873a05be12ce, State: Running, Role: LEADER
I20260925 10:32:15.285882 2654 consensus_queue.cc:237] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:15.286164 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65f987f616b413e8453428bba31234b" candidate_uuid: "98b2ae42644544c3ba0d44d87e02e76a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4182079abb4ae0bfd02ce5d1e141df"
I20260925 10:32:15.286175 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65f987f616b413e8453428bba31234b" candidate_uuid: "98b2ae42644544c3ba0d44d87e02e76a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0729ceedca9436d8691873a05be12ce"
I20260925 10:32:15.286433 2304 raft_consensus.cc:2379] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 98b2ae42644544c3ba0d44d87e02e76a in current term 1: Already voted for candidate b0729ceedca9436d8691873a05be12ce in this term.
I20260925 10:32:15.287037 2512 leader_election.cc:302] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98b2ae42644544c3ba0d44d87e02e76a; no voters: b0729ceedca9436d8691873a05be12ce, ce4182079abb4ae0bfd02ce5d1e141df
I20260925 10:32:15.287393 2653 raft_consensus.cc:2726] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 10:32:15.287683 2658 tablet_replica.cc:439] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce: TxnStatusTablet state changed. Reason: New leader b0729ceedca9436d8691873a05be12ce. Latest consensus state: current_term: 1 leader_uuid: "b0729ceedca9436d8691873a05be12ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } }
I20260925 10:32:15.288070 2658 tablet_replica.cc:442] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:32:15.288797 2673 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:32:15.289093 2673 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:32:15.289318 2143 catalog_manager.cc:5495] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce reported cstate change: term changed from 0 to 1, leader changed from <none> to b0729ceedca9436d8691873a05be12ce (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b0729ceedca9436d8691873a05be12ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } health_report { overall_health: UNKNOWN } } }
I20260925 10:32:15.317765 2658 consensus_queue.cc:1035] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:15.325843 2654 consensus_queue.cc:1035] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:15.362210 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:15.356460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.0.97.2:44639)
0925 10:32:15.356655 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:15.356661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:15.356681 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:32:15.357005 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:15.357011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:15.357107 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:15.357257 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:32:15.357560 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.358788 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.360037 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.360454 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.360632 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:15.361073 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:32:15.361588 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:32:15.361769 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:15.361817 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 10:32:15.361899 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:32:15.362219 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:15.356364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44639 (local address 127.0.0.1:41778)
0925 10:32:15.356603 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:15.356620 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:15.356782 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:15.357283 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:15.357286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:15.357296 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:32:15.357439 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:15.357446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.358922 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.358926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:15.359875 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:15.359886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.360643 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.360648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:15.360805 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:15.360867 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:32:15.361701 (+ 834us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:32:15.361720 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 10:32:15.361793 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 10:32:15.361930 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 10:32:15.369027 2304 raft_consensus.cc:1261] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Refusing update from remote peer b0729ceedca9436d8691873a05be12ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:15.369071 2578 raft_consensus.cc:1261] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Refusing update from remote peer b0729ceedca9436d8691873a05be12ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:15.369630 2654 consensus_queue.cc:1035] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:15.369936 2658 consensus_queue.cc:1035] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:15.376576 2653 tablet_replica.cc:439] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a: TxnStatusTablet state changed. Reason: New leader b0729ceedca9436d8691873a05be12ce. Latest consensus state: current_term: 1 leader_uuid: "b0729ceedca9436d8691873a05be12ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } }
I20260925 10:32:15.376905 2653 tablet_replica.cc:442] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:32:15.379694 2658 tablet_replica.cc:439] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0729ceedca9436d8691873a05be12ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } }
I20260925 10:32:15.379912 2658 tablet_replica.cc:442] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:32:15.380507 2679 mvcc.cc:205] Tried to move back new op lower bound from 7333201245664739328 to 7333201245333852160. Current Snapshot: MvccSnapshot[applied={T|T < 7333201245664739328}]
I20260925 10:32:15.380908 2653 tablet_replica.cc:439] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0729ceedca9436d8691873a05be12ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } }
I20260925 10:32:15.381145 2653 tablet_replica.cc:442] T c65f987f616b413e8453428bba31234b P 98b2ae42644544c3ba0d44d87e02e76a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:32:15.391433 2654 tablet_replica.cc:439] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0729ceedca9436d8691873a05be12ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } }
I20260925 10:32:15.391732 2654 tablet_replica.cc:442] T c65f987f616b413e8453428bba31234b P b0729ceedca9436d8691873a05be12ce: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:32:15.393224 2655 tablet_replica.cc:439] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df: TxnStatusTablet state changed. Reason: New leader b0729ceedca9436d8691873a05be12ce. Latest consensus state: current_term: 1 leader_uuid: "b0729ceedca9436d8691873a05be12ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } }
I20260925 10:32:15.393491 2655 tablet_replica.cc:442] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:32:15.396957 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:15.389545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41788 (local address 127.0.97.2:44639)
0925 10:32:15.389713 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:15.389719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:15.389952 (+ 233us) server_negotiation.cc:407] Connection header received
0925 10:32:15.390232 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:15.390241 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:15.390382 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:15.390457 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:32:15.391114 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.392440 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.394492 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.395091 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.395208 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:15.395898 (+ 690us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:32:15.396071 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:32:15.396151 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:15.396433 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 10:32:15.396540 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:32:15.400790 2655 tablet_replica.cc:439] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0729ceedca9436d8691873a05be12ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } } }
I20260925 10:32:15.401011 2655 tablet_replica.cc:442] T c65f987f616b413e8453428bba31234b P ce4182079abb4ae0bfd02ce5d1e141df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:32:15.415205 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:15.408454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43117 (local address 127.0.97.2:44639)
0925 10:32:15.408663 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:15.408669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:15.409325 (+ 656us) server_negotiation.cc:407] Connection header received
0925 10:32:15.409504 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:15.409510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:15.409585 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:15.409679 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:32:15.410096 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.411307 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.413431 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.413886 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.413976 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:15.414525 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:32:15.414613 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:32:15.414679 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:15.414853 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 10:32:15.414941 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":56}
I20260925 10:32:15.415524 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:15.408530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44639 (local address 127.0.97.2:43117)
0925 10:32:15.409211 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:15.409240 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:15.409384 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:15.409703 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:15.409706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:15.409715 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:32:15.409949 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:15.409956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.411451 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.411457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:15.413259 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:15.413295 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:15.414034 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:15.414041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:15.414218 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:15.414291 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:32:15.414709 (+ 418us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:32:15.414720 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 10:32:15.414813 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 10:32:15.414982 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":1,"thread_start_us":344,"threads_started":1}
I20260925 10:32:15.513644 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:15.513906 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:15.517887 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e: Sending DeleteTablet for 3 replicas of tablet d67c4bafadb54da5aad75beb2f1dbdf7
I20260925 10:32:15.519021 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet d67c4bafadb54da5aad75beb2f1dbdf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54714
I20260925 10:32:15.519865 2695 tablet_replica.cc:330] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a: stopping tablet replica
I20260925 10:32:15.519775 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet d67c4bafadb54da5aad75beb2f1dbdf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41774
I20260925 10:32:15.520388 2695 raft_consensus.cc:2229] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:15.520762 2695 raft_consensus.cc:2258] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:15.520738 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet d67c4bafadb54da5aad75beb2f1dbdf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44522
I20260925 10:32:15.522610 2695 ts_tablet_manager.cc:1905] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:15.523453 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:32:15.524070 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:32:15.532264 2419 tablet_service.cc:1467] Processing CreateTablet for tablet 56c878b4413f4009abb12ad382233a72 (DEFAULT_TABLE table=test-table [id=09c2975520f34b79b5cef94c5b84af08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:15.532677 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c878b4413f4009abb12ad382233a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:15.533087 2279 tablet_service.cc:1467] Processing CreateTablet for tablet 56c878b4413f4009abb12ad382233a72 (DEFAULT_TABLE table=test-table [id=09c2975520f34b79b5cef94c5b84af08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:15.533654 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c878b4413f4009abb12ad382233a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:15.538715 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 56c878b4413f4009abb12ad382233a72 (DEFAULT_TABLE table=test-table [id=09c2975520f34b79b5cef94c5b84af08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:15.539196 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c878b4413f4009abb12ad382233a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:15.550787 2698 tablet_bootstrap.cc:492] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce: Bootstrap starting.
I20260925 10:32:15.552076 2695 ts_tablet_manager.cc:1918] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:32:15.552290 2695 log.cc:1196] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/d67c4bafadb54da5aad75beb2f1dbdf7
I20260925 10:32:15.552680 2698 tablet_bootstrap.cc:654] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:15.553030 2695 ts_tablet_manager.cc:1939] T d67c4bafadb54da5aad75beb2f1dbdf7 P 98b2ae42644544c3ba0d44d87e02e76a: Deleting consensus metadata
I20260925 10:32:15.555809 2131 catalog_manager.cc:4841] TS 98b2ae42644544c3ba0d44d87e02e76a (127.0.97.3:33975): tablet d67c4bafadb54da5aad75beb2f1dbdf7 (table test-table [id=eb34fd366e1b4296a9b0afd7de7d87be]) successfully deleted
I20260925 10:32:15.565300 2698 tablet_bootstrap.cc:492] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce: No bootstrap required, opened a new log
I20260925 10:32:15.565464 2698 ts_tablet_manager.cc:1397] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.006s
I20260925 10:32:15.566519 2698 raft_consensus.cc:348] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:15.566871 2698 raft_consensus.cc:374] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:15.566968 2698 raft_consensus.cc:729] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0729ceedca9436d8691873a05be12ce, State: Initialized, Role: FOLLOWER
I20260925 10:32:15.567237 2698 consensus_queue.cc:260] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:15.567822 2698 ts_tablet_manager.cc:1428] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:32:15.571005 2699 tablet_bootstrap.cc:492] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a: Bootstrap starting.
I20260925 10:32:15.573084 2699 tablet_bootstrap.cc:654] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:15.574523 2701 tablet_bootstrap.cc:492] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df: Bootstrap starting.
I20260925 10:32:15.576565 2697 tablet_replica.cc:330] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df: stopping tablet replica
I20260925 10:32:15.576725 2701 tablet_bootstrap.cc:654] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:15.577203 2696 tablet_replica.cc:330] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce: stopping tablet replica
I20260925 10:32:15.577643 2697 raft_consensus.cc:2229] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:15.578069 2697 raft_consensus.cc:2258] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:15.578565 2696 raft_consensus.cc:2229] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:32:15.579393 2696 raft_consensus.cc:2258] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:15.580256 2699 tablet_bootstrap.cc:492] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a: No bootstrap required, opened a new log
I20260925 10:32:15.580427 2699 ts_tablet_manager.cc:1397] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260925 10:32:15.581139 2696 ts_tablet_manager.cc:1905] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:15.581260 2699 raft_consensus.cc:348] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:15.581548 2699 raft_consensus.cc:374] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:15.581650 2699 raft_consensus.cc:729] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98b2ae42644544c3ba0d44d87e02e76a, State: Initialized, Role: FOLLOWER
I20260925 10:32:15.583446 2697 ts_tablet_manager.cc:1905] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:15.588377 2699 consensus_queue.cc:260] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:15.589181 2699 ts_tablet_manager.cc:1428] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20260925 10:32:15.590345 2701 tablet_bootstrap.cc:492] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df: No bootstrap required, opened a new log
I20260925 10:32:15.590343 2696 ts_tablet_manager.cc:1918] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:32:15.590499 2701 ts_tablet_manager.cc:1397] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20260925 10:32:15.590660 2696 log.cc:1196] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/d67c4bafadb54da5aad75beb2f1dbdf7
I20260925 10:32:15.591245 2696 ts_tablet_manager.cc:1939] T d67c4bafadb54da5aad75beb2f1dbdf7 P b0729ceedca9436d8691873a05be12ce: Deleting consensus metadata
I20260925 10:32:15.591280 2701 raft_consensus.cc:348] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:15.591580 2701 raft_consensus.cc:374] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:15.591686 2701 raft_consensus.cc:729] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4182079abb4ae0bfd02ce5d1e141df, State: Initialized, Role: FOLLOWER
I20260925 10:32:15.591954 2701 consensus_queue.cc:260] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:15.593152 2128 catalog_manager.cc:4841] TS b0729ceedca9436d8691873a05be12ce (127.0.97.2:44639): tablet d67c4bafadb54da5aad75beb2f1dbdf7 (table test-table [id=eb34fd366e1b4296a9b0afd7de7d87be]) successfully deleted
I20260925 10:32:15.594681 2701 ts_tablet_manager.cc:1428] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260925 10:32:15.601133 2697 ts_tablet_manager.cc:1918] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:32:15.601312 2697 log.cc:1196] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/d67c4bafadb54da5aad75beb2f1dbdf7
I20260925 10:32:15.601850 2697 ts_tablet_manager.cc:1939] T d67c4bafadb54da5aad75beb2f1dbdf7 P ce4182079abb4ae0bfd02ce5d1e141df: Deleting consensus metadata
I20260925 10:32:15.603776 2131 catalog_manager.cc:4841] TS ce4182079abb4ae0bfd02ce5d1e141df (127.0.97.1:41351): tablet d67c4bafadb54da5aad75beb2f1dbdf7 (table test-table [id=eb34fd366e1b4296a9b0afd7de7d87be]) successfully deleted
I20260925 10:32:15.630541 2692 raft_consensus.cc:482] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:32:15.630733 2692 raft_consensus.cc:504] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:15.631527 2692 leader_election.cc:288] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce4182079abb4ae0bfd02ce5d1e141df (127.0.97.1:41351), 98b2ae42644544c3ba0d44d87e02e76a (127.0.97.3:33975)
I20260925 10:32:15.632251 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c878b4413f4009abb12ad382233a72" candidate_uuid: "b0729ceedca9436d8691873a05be12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98b2ae42644544c3ba0d44d87e02e76a" is_pre_election: true
I20260925 10:32:15.632593 2577 raft_consensus.cc:2454] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0729ceedca9436d8691873a05be12ce in term 0.
I20260925 10:32:15.633234 2303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c878b4413f4009abb12ad382233a72" candidate_uuid: "b0729ceedca9436d8691873a05be12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" is_pre_election: true
I20260925 10:32:15.633574 2303 raft_consensus.cc:2454] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0729ceedca9436d8691873a05be12ce in term 0.
I20260925 10:32:15.634296 2373 leader_election.cc:302] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98b2ae42644544c3ba0d44d87e02e76a, b0729ceedca9436d8691873a05be12ce; no voters:
I20260925 10:32:15.635283 2692 raft_consensus.cc:2781] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:32:15.635411 2692 raft_consensus.cc:482] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:32:15.635515 2692 raft_consensus.cc:3037] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:15.641063 2692 raft_consensus.cc:504] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:15.642484 2303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c878b4413f4009abb12ad382233a72" candidate_uuid: "b0729ceedca9436d8691873a05be12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4182079abb4ae0bfd02ce5d1e141df"
I20260925 10:32:15.642730 2303 raft_consensus.cc:3037] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:15.646214 2303 raft_consensus.cc:2454] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0729ceedca9436d8691873a05be12ce in term 1.
I20260925 10:32:15.646754 2373 leader_election.cc:302] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0729ceedca9436d8691873a05be12ce, ce4182079abb4ae0bfd02ce5d1e141df; no voters:
I20260925 10:32:15.647127 2687 raft_consensus.cc:2781] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:15.647320 2687 raft_consensus.cc:686] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [term 1 LEADER]: Becoming Leader. State: Replica: b0729ceedca9436d8691873a05be12ce, State: Running, Role: LEADER
I20260925 10:32:15.647663 2687 consensus_queue.cc:237] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:15.650468 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c878b4413f4009abb12ad382233a72" candidate_uuid: "b0729ceedca9436d8691873a05be12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98b2ae42644544c3ba0d44d87e02e76a"
I20260925 10:32:15.650707 2577 raft_consensus.cc:3037] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:15.654409 2577 raft_consensus.cc:2454] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0729ceedca9436d8691873a05be12ce in term 1.
I20260925 10:32:15.655191 2692 leader_election.cc:288] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [CANDIDATE]: Term 1 election: Requested vote from peers ce4182079abb4ae0bfd02ce5d1e141df (127.0.97.1:41351), 98b2ae42644544c3ba0d44d87e02e76a (127.0.97.3:33975)
I20260925 10:32:15.664744 2143 catalog_manager.cc:5495] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce reported cstate change: term changed from 0 to 1, leader changed from <none> to b0729ceedca9436d8691873a05be12ce (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b0729ceedca9436d8691873a05be12ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } health_report { overall_health: UNKNOWN } } }
I20260925 10:32:15.724745 2303 raft_consensus.cc:1261] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Refusing update from remote peer b0729ceedca9436d8691873a05be12ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:15.724761 2577 raft_consensus.cc:1261] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Refusing update from remote peer b0729ceedca9436d8691873a05be12ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:15.725292 2654 consensus_queue.cc:1035] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:15.725847 2658 consensus_queue.cc:1035] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:15.738245 2676 mvcc.cc:205] Tried to move back new op lower bound from 7333201247121920000 to 7333201246816280576. Current Snapshot: MvccSnapshot[applied={T|T < 7333201247121920000}]
W20260925 10:32:15.764887 2676 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 10:32:17.342908 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:17.343544 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:17.347427 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e: Sending DeleteTablet for 3 replicas of tablet 56c878b4413f4009abb12ad382233a72
I20260925 10:32:17.348141 2279 tablet_service.cc:1514] Processing DeleteTablet for tablet 56c878b4413f4009abb12ad382233a72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44522
I20260925 10:32:17.348212 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet 56c878b4413f4009abb12ad382233a72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41774
I20260925 10:32:17.348345 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 56c878b4413f4009abb12ad382233a72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54714
I20260925 10:32:17.359336 2728 tablet_replica.cc:330] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a: stopping tablet replica
I20260925 10:32:17.359550 2728 raft_consensus.cc:2229] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:17.359781 2728 raft_consensus.cc:2258] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:17.360272 2726 tablet_replica.cc:330] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df: stopping tablet replica
I20260925 10:32:17.360476 2726 raft_consensus.cc:2229] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:17.360714 2726 raft_consensus.cc:2258] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:17.360996 2728 ts_tablet_manager.cc:1905] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:17.361239 2727 tablet_replica.cc:330] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce: stopping tablet replica
I20260925 10:32:17.361418 2727 raft_consensus.cc:2229] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:32:17.361675 2726 ts_tablet_manager.cc:1905] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:17.361786 2727 raft_consensus.cc:2258] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:17.362717 2727 ts_tablet_manager.cc:1905] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:17.367915 2728 ts_tablet_manager.cc:1918] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:32:17.368091 2728 log.cc:1196] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/56c878b4413f4009abb12ad382233a72
I20260925 10:32:17.368407 2726 ts_tablet_manager.cc:1918] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:32:17.368573 2728 ts_tablet_manager.cc:1939] T 56c878b4413f4009abb12ad382233a72 P 98b2ae42644544c3ba0d44d87e02e76a: Deleting consensus metadata
I20260925 10:32:17.368572 2726 log.cc:1196] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/56c878b4413f4009abb12ad382233a72
I20260925 10:32:17.368981 2726 ts_tablet_manager.cc:1939] T 56c878b4413f4009abb12ad382233a72 P ce4182079abb4ae0bfd02ce5d1e141df: Deleting consensus metadata
I20260925 10:32:17.369289 2727 ts_tablet_manager.cc:1918] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:32:17.369442 2727 log.cc:1196] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/56c878b4413f4009abb12ad382233a72
I20260925 10:32:17.369822 2131 catalog_manager.cc:4841] TS 98b2ae42644544c3ba0d44d87e02e76a (127.0.97.3:33975): tablet 56c878b4413f4009abb12ad382233a72 (table test-table [id=09c2975520f34b79b5cef94c5b84af08]) successfully deleted
I20260925 10:32:17.369920 2727 ts_tablet_manager.cc:1939] T 56c878b4413f4009abb12ad382233a72 P b0729ceedca9436d8691873a05be12ce: Deleting consensus metadata
I20260925 10:32:17.370100 2131 catalog_manager.cc:4841] TS ce4182079abb4ae0bfd02ce5d1e141df (127.0.97.1:41351): tablet 56c878b4413f4009abb12ad382233a72 (table test-table [id=09c2975520f34b79b5cef94c5b84af08]) successfully deleted
I20260925 10:32:17.370954 2128 catalog_manager.cc:4841] TS b0729ceedca9436d8691873a05be12ce (127.0.97.2:44639): tablet 56c878b4413f4009abb12ad382233a72 (table test-table [id=09c2975520f34b79b5cef94c5b84af08]) successfully deleted
Sep 25 10:32:17 dist-test-slave-dxdd krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:32:17 dist-test-slave-dxdd krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:17.403733 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:17.390378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51950 (local address 127.0.97.62:34799)
0925 10:32:17.390927 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:17.390931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:17.391029 (+ 98us) server_negotiation.cc:407] Connection header received
0925 10:32:17.391159 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:17.391163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:17.391233 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:17.391328 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:17.392809 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.393560 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.394751 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.395140 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.395228 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:17.398889 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:17.398894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:17.398897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:17.398926 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:17.401241 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:17.401899 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:17.401922 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:17.401927 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:17.402072 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:17.402475 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:17.402479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:17.402481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:17.402727 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:17.402916 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:17.403221 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 10:32:17.403386 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":314,"threads_started":1}
I20260925 10:32:17.407577 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:32:17.408171 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:32:17.413904 2279 tablet_service.cc:1467] Processing CreateTablet for tablet b4fc65a221584046991614e34cb9e881 (DEFAULT_TABLE table=test-table [id=b0682f435c2146009749b661f987b687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:17.414021 2419 tablet_service.cc:1467] Processing CreateTablet for tablet b4fc65a221584046991614e34cb9e881 (DEFAULT_TABLE table=test-table [id=b0682f435c2146009749b661f987b687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:17.414386 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4fc65a221584046991614e34cb9e881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:17.414271 2558 tablet_service.cc:1467] Processing CreateTablet for tablet b4fc65a221584046991614e34cb9e881 (DEFAULT_TABLE table=test-table [id=b0682f435c2146009749b661f987b687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:17.414443 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4fc65a221584046991614e34cb9e881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:17.414633 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4fc65a221584046991614e34cb9e881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:17.419142 2737 tablet_bootstrap.cc:492] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a: Bootstrap starting.
I20260925 10:32:17.420601 2737 tablet_bootstrap.cc:654] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:17.420804 2738 tablet_bootstrap.cc:492] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce: Bootstrap starting.
I20260925 10:32:17.421635 2739 tablet_bootstrap.cc:492] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df: Bootstrap starting.
I20260925 10:32:17.423084 2737 tablet_bootstrap.cc:492] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a: No bootstrap required, opened a new log
I20260925 10:32:17.423169 2739 tablet_bootstrap.cc:654] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:17.423254 2737 ts_tablet_manager.cc:1397] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260925 10:32:17.423616 2738 tablet_bootstrap.cc:654] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:17.424021 2737 raft_consensus.cc:348] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.424340 2737 raft_consensus.cc:374] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:17.424481 2737 raft_consensus.cc:729] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98b2ae42644544c3ba0d44d87e02e76a, State: Initialized, Role: FOLLOWER
I20260925 10:32:17.424788 2737 consensus_queue.cc:260] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.425420 2739 tablet_bootstrap.cc:492] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df: No bootstrap required, opened a new log
I20260925 10:32:17.425563 2739 ts_tablet_manager.cc:1397] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 10:32:17.426088 2739 raft_consensus.cc:348] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.426280 2739 raft_consensus.cc:374] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:17.426422 2739 raft_consensus.cc:729] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4182079abb4ae0bfd02ce5d1e141df, State: Initialized, Role: FOLLOWER
I20260925 10:32:17.426656 2739 consensus_queue.cc:260] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.428737 2739 ts_tablet_manager.cc:1428] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:32:17.431865 2738 tablet_bootstrap.cc:492] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce: No bootstrap required, opened a new log
I20260925 10:32:17.432006 2738 ts_tablet_manager.cc:1397] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260925 10:32:17.432811 2738 raft_consensus.cc:348] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.433023 2738 raft_consensus.cc:374] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:17.433113 2738 raft_consensus.cc:729] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0729ceedca9436d8691873a05be12ce, State: Initialized, Role: FOLLOWER
I20260925 10:32:17.433315 2738 consensus_queue.cc:260] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.433797 2738 ts_tablet_manager.cc:1428] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 10:32:17.434144 2737 ts_tablet_manager.cc:1428] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a: Time spent starting tablet: real 0.011s user 0.001s sys 0.005s
I20260925 10:32:17.473487 2687 raft_consensus.cc:482] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:32:17.473657 2687 raft_consensus.cc:504] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.474135 2687 leader_election.cc:288] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce4182079abb4ae0bfd02ce5d1e141df (127.0.97.1:41351), 98b2ae42644544c3ba0d44d87e02e76a (127.0.97.3:33975)
I20260925 10:32:17.474723 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4fc65a221584046991614e34cb9e881" candidate_uuid: "b0729ceedca9436d8691873a05be12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" is_pre_election: true
I20260925 10:32:17.474790 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4fc65a221584046991614e34cb9e881" candidate_uuid: "b0729ceedca9436d8691873a05be12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98b2ae42644544c3ba0d44d87e02e76a" is_pre_election: true
I20260925 10:32:17.474977 2304 raft_consensus.cc:2454] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0729ceedca9436d8691873a05be12ce in term 0.
I20260925 10:32:17.475013 2577 raft_consensus.cc:2454] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0729ceedca9436d8691873a05be12ce in term 0.
I20260925 10:32:17.475446 2373 leader_election.cc:302] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98b2ae42644544c3ba0d44d87e02e76a, b0729ceedca9436d8691873a05be12ce; no voters:
I20260925 10:32:17.475731 2687 raft_consensus.cc:2781] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:32:17.475809 2687 raft_consensus.cc:482] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:32:17.475919 2687 raft_consensus.cc:3037] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:17.477829 2687 raft_consensus.cc:504] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.478418 2687 leader_election.cc:288] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [CANDIDATE]: Term 1 election: Requested vote from peers ce4182079abb4ae0bfd02ce5d1e141df (127.0.97.1:41351), 98b2ae42644544c3ba0d44d87e02e76a (127.0.97.3:33975)
I20260925 10:32:17.478793 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4fc65a221584046991614e34cb9e881" candidate_uuid: "b0729ceedca9436d8691873a05be12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4182079abb4ae0bfd02ce5d1e141df"
I20260925 10:32:17.478910 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4fc65a221584046991614e34cb9e881" candidate_uuid: "b0729ceedca9436d8691873a05be12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98b2ae42644544c3ba0d44d87e02e76a"
I20260925 10:32:17.479010 2304 raft_consensus.cc:3037] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:17.479090 2577 raft_consensus.cc:3037] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:17.481829 2304 raft_consensus.cc:2454] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0729ceedca9436d8691873a05be12ce in term 1.
I20260925 10:32:17.481861 2577 raft_consensus.cc:2454] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0729ceedca9436d8691873a05be12ce in term 1.
I20260925 10:32:17.482182 2373 leader_election.cc:302] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0729ceedca9436d8691873a05be12ce, ce4182079abb4ae0bfd02ce5d1e141df; no voters:
I20260925 10:32:17.482436 2687 raft_consensus.cc:2781] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:17.482596 2687 raft_consensus.cc:686] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [term 1 LEADER]: Becoming Leader. State: Replica: b0729ceedca9436d8691873a05be12ce, State: Running, Role: LEADER
I20260925 10:32:17.482846 2687 consensus_queue.cc:237] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.485081 2143 catalog_manager.cc:5495] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce reported cstate change: term changed from 0 to 1, leader changed from <none> to b0729ceedca9436d8691873a05be12ce (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b0729ceedca9436d8691873a05be12ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } health_report { overall_health: UNKNOWN } } }
I20260925 10:32:17.512730 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:17.508221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41792 (local address 127.0.97.2:44639)
0925 10:32:17.508685 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:17.508690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:17.508723 (+ 33us) server_negotiation.cc:407] Connection header received
0925 10:32:17.508767 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:17.508771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:17.508845 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:17.508918 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:32:17.509211 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.509940 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.511226 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.511608 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.511692 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:17.512063 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:32:17.512149 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:32:17.512243 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:17.512368 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 10:32:17.512419 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":284,"threads_started":1}
I20260925 10:32:17.520026 2577 raft_consensus.cc:1261] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Refusing update from remote peer b0729ceedca9436d8691873a05be12ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:17.520092 2303 raft_consensus.cc:1261] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Refusing update from remote peer b0729ceedca9436d8691873a05be12ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:17.520571 2714 consensus_queue.cc:1035] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:17.520879 2687 consensus_queue.cc:1035] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:17.529747 2711 mvcc.cc:205] Tried to move back new op lower bound from 7333201254476828672 to 7333201254331555840. Current Snapshot: MvccSnapshot[applied={T|T < 7333201254476828672}]
I20260925 10:32:17.559167 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:17.559345 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:17.561851 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e: Sending DeleteTablet for 3 replicas of tablet b4fc65a221584046991614e34cb9e881
I20260925 10:32:17.562467 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet b4fc65a221584046991614e34cb9e881 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41774
I20260925 10:32:17.562552 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet b4fc65a221584046991614e34cb9e881 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54714
I20260925 10:32:17.562561 2279 tablet_service.cc:1514] Processing DeleteTablet for tablet b4fc65a221584046991614e34cb9e881 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44522
I20260925 10:32:17.562803 2728 tablet_replica.cc:330] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a: stopping tablet replica
I20260925 10:32:17.562804 2727 tablet_replica.cc:330] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce: stopping tablet replica
I20260925 10:32:17.562997 2728 raft_consensus.cc:2229] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:17.563010 2727 raft_consensus.cc:2229] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:32:17.563192 2728 raft_consensus.cc:2258] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:17.563297 2727 raft_consensus.cc:2258] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:17.563264 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 10:32:17.563526 2726 tablet_replica.cc:330] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df: stopping tablet replica
I20260925 10:32:17.563692 2726 raft_consensus.cc:2229] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:17.563850 2726 raft_consensus.cc:2258] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 10:32:17.563879 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:32:17.564254 2728 ts_tablet_manager.cc:1905] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:17.564667 2726 ts_tablet_manager.cc:1905] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:17.564862 2727 ts_tablet_manager.cc:1905] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:17.570529 2558 tablet_service.cc:1467] Processing CreateTablet for tablet f22e2ecd501546afbdba7532c09e6469 (DEFAULT_TABLE table=test-table [id=b417cd0df94345b682fb99b371564738]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:17.570811 2727 ts_tablet_manager.cc:1918] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:32:17.570943 2727 log.cc:1196] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/b4fc65a221584046991614e34cb9e881
I20260925 10:32:17.570986 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22e2ecd501546afbdba7532c09e6469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:17.571010 2728 ts_tablet_manager.cc:1918] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:32:17.571158 2728 log.cc:1196] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/b4fc65a221584046991614e34cb9e881
I20260925 10:32:17.571261 2726 ts_tablet_manager.cc:1918] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:32:17.571372 2727 ts_tablet_manager.cc:1939] T b4fc65a221584046991614e34cb9e881 P b0729ceedca9436d8691873a05be12ce: Deleting consensus metadata
I20260925 10:32:17.571468 2726 log.cc:1196] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/b4fc65a221584046991614e34cb9e881
I20260925 10:32:17.571626 2728 ts_tablet_manager.cc:1939] T b4fc65a221584046991614e34cb9e881 P 98b2ae42644544c3ba0d44d87e02e76a: Deleting consensus metadata
I20260925 10:32:17.571935 2726 ts_tablet_manager.cc:1939] T b4fc65a221584046991614e34cb9e881 P ce4182079abb4ae0bfd02ce5d1e141df: Deleting consensus metadata
I20260925 10:32:17.572636 2279 tablet_service.cc:1467] Processing CreateTablet for tablet f22e2ecd501546afbdba7532c09e6469 (DEFAULT_TABLE table=test-table [id=b417cd0df94345b682fb99b371564738]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:17.573009 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22e2ecd501546afbdba7532c09e6469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:17.574559 2131 catalog_manager.cc:4841] TS 98b2ae42644544c3ba0d44d87e02e76a (127.0.97.3:33975): tablet b4fc65a221584046991614e34cb9e881 (table test-table [id=b0682f435c2146009749b661f987b687]) successfully deleted
I20260925 10:32:17.574615 2128 catalog_manager.cc:4841] TS b0729ceedca9436d8691873a05be12ce (127.0.97.2:44639): tablet b4fc65a221584046991614e34cb9e881 (table test-table [id=b0682f435c2146009749b661f987b687]) successfully deleted
I20260925 10:32:17.575059 2418 tablet_service.cc:1467] Processing CreateTablet for tablet f22e2ecd501546afbdba7532c09e6469 (DEFAULT_TABLE table=test-table [id=b417cd0df94345b682fb99b371564738]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:17.575415 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22e2ecd501546afbdba7532c09e6469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:17.578114 2737 tablet_bootstrap.cc:492] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a: Bootstrap starting.
I20260925 10:32:17.579079 2739 tablet_bootstrap.cc:492] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df: Bootstrap starting.
I20260925 10:32:17.579155 2131 catalog_manager.cc:4841] TS ce4182079abb4ae0bfd02ce5d1e141df (127.0.97.1:41351): tablet b4fc65a221584046991614e34cb9e881 (table test-table [id=b0682f435c2146009749b661f987b687]) successfully deleted
I20260925 10:32:17.579643 2737 tablet_bootstrap.cc:654] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:17.580523 2739 tablet_bootstrap.cc:654] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:17.581386 2737 tablet_bootstrap.cc:492] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a: No bootstrap required, opened a new log
I20260925 10:32:17.581385 2738 tablet_bootstrap.cc:492] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce: Bootstrap starting.
I20260925 10:32:17.581547 2737 ts_tablet_manager.cc:1397] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:32:17.582062 2737 raft_consensus.cc:348] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.582267 2737 raft_consensus.cc:374] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:17.582383 2737 raft_consensus.cc:729] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98b2ae42644544c3ba0d44d87e02e76a, State: Initialized, Role: FOLLOWER
I20260925 10:32:17.582646 2737 consensus_queue.cc:260] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.583192 2737 ts_tablet_manager.cc:1428] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:32:17.583240 2738 tablet_bootstrap.cc:654] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:17.583560 2739 tablet_bootstrap.cc:492] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df: No bootstrap required, opened a new log
I20260925 10:32:17.583693 2739 ts_tablet_manager.cc:1397] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 10:32:17.584398 2739 raft_consensus.cc:348] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.584632 2739 raft_consensus.cc:374] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:17.584718 2739 raft_consensus.cc:729] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4182079abb4ae0bfd02ce5d1e141df, State: Initialized, Role: FOLLOWER
I20260925 10:32:17.584937 2739 consensus_queue.cc:260] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.585156 2738 tablet_bootstrap.cc:492] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce: No bootstrap required, opened a new log
I20260925 10:32:17.585311 2738 ts_tablet_manager.cc:1397] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 10:32:17.585724 2739 ts_tablet_manager.cc:1428] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:32:17.585985 2738 raft_consensus.cc:348] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.586215 2738 raft_consensus.cc:374] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:17.586295 2738 raft_consensus.cc:729] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0729ceedca9436d8691873a05be12ce, State: Initialized, Role: FOLLOWER
I20260925 10:32:17.586561 2738 consensus_queue.cc:260] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.587386 2738 ts_tablet_manager.cc:1428] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:32:17.738164 2744 raft_consensus.cc:482] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:32:17.738428 2744 raft_consensus.cc:504] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.740032 2744 leader_election.cc:288] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0729ceedca9436d8691873a05be12ce (127.0.97.2:44639), 98b2ae42644544c3ba0d44d87e02e76a (127.0.97.3:33975)
I20260925 10:32:17.749073 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:17.740398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39127 (local address 127.0.97.2:44639)
0925 10:32:17.740567 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:17.740573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:17.742180 (+ 1607us) server_negotiation.cc:407] Connection header received
0925 10:32:17.742331 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:17.742337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:17.742421 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:17.742513 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:17.743061 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.744026 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.746762 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.748013 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.748150 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:17.748247 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:17.748698 (+ 451us) server_negotiation.cc:299] Negotiation successful
0925 10:32:17.748819 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:32:17.749282 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:17.740389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33975 (local address 127.0.97.1:45627)
0925 10:32:17.741745 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:17.741765 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:17.741946 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:17.742363 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:17.742369 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:17.742392 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:17.742732 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:17.742744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.744157 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.744162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:17.746622 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:17.746631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.748214 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.748225 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:17.748495 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:17.748569 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 10:32:17.748668 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:32:17.748823 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1238,"mutex_wait_us":291,"thread_start_us":1091,"threads_started":1}
I20260925 10:32:17.749305 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:17.740590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44639 (local address 127.0.97.1:39127)
0925 10:32:17.742076 (+ 1486us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:17.742094 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:17.742200 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:17.742539 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:17.742547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:17.742564 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:17.742913 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:17.742922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.744160 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.744163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:17.746618 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:17.746631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.748240 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.748245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:17.748490 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:17.748569 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 10:32:17.748665 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 10:32:17.748823 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1380,"mutex_wait_us":287,"thread_start_us":1217,"threads_started":1}
I20260925 10:32:17.749123 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:17.740391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45627 (local address 127.0.97.3:33975)
0925 10:32:17.741579 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:17.741586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:17.741860 (+ 274us) server_negotiation.cc:407] Connection header received
0925 10:32:17.742104 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:17.742108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:17.742208 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:17.742333 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:17.742910 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.744010 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.746762 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.748013 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.748145 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:17.748264 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:17.748698 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 10:32:17.748822 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1016,"thread_start_us":835,"threads_started":1}
I20260925 10:32:17.749954 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22e2ecd501546afbdba7532c09e6469" candidate_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0729ceedca9436d8691873a05be12ce" is_pre_election: true
I20260925 10:32:17.749974 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22e2ecd501546afbdba7532c09e6469" candidate_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98b2ae42644544c3ba0d44d87e02e76a" is_pre_election: true
I20260925 10:32:17.750208 2577 raft_consensus.cc:2454] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce4182079abb4ae0bfd02ce5d1e141df in term 0.
I20260925 10:32:17.750236 2444 raft_consensus.cc:2454] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce4182079abb4ae0bfd02ce5d1e141df in term 0.
I20260925 10:32:17.750684 2234 leader_election.cc:302] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 98b2ae42644544c3ba0d44d87e02e76a, b0729ceedca9436d8691873a05be12ce, ce4182079abb4ae0bfd02ce5d1e141df; no voters:
I20260925 10:32:17.751082 2744 raft_consensus.cc:2781] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:32:17.751199 2744 raft_consensus.cc:482] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:32:17.751315 2744 raft_consensus.cc:3037] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:17.753693 2744 raft_consensus.cc:504] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.754216 2744 leader_election.cc:288] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [CANDIDATE]: Term 1 election: Requested vote from peers b0729ceedca9436d8691873a05be12ce (127.0.97.2:44639), 98b2ae42644544c3ba0d44d87e02e76a (127.0.97.3:33975)
I20260925 10:32:17.754629 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22e2ecd501546afbdba7532c09e6469" candidate_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0729ceedca9436d8691873a05be12ce"
I20260925 10:32:17.754849 2444 raft_consensus.cc:3037] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:17.754802 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22e2ecd501546afbdba7532c09e6469" candidate_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98b2ae42644544c3ba0d44d87e02e76a"
I20260925 10:32:17.754982 2577 raft_consensus.cc:3037] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:17.757663 2577 raft_consensus.cc:2454] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce4182079abb4ae0bfd02ce5d1e141df in term 1.
I20260925 10:32:17.757659 2444 raft_consensus.cc:2454] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce4182079abb4ae0bfd02ce5d1e141df in term 1.
I20260925 10:32:17.757997 2231 leader_election.cc:302] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0729ceedca9436d8691873a05be12ce, ce4182079abb4ae0bfd02ce5d1e141df; no voters:
I20260925 10:32:17.758255 2744 raft_consensus.cc:2781] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:17.758668 2744 raft_consensus.cc:686] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [term 1 LEADER]: Becoming Leader. State: Replica: ce4182079abb4ae0bfd02ce5d1e141df, State: Running, Role: LEADER
I20260925 10:32:17.759014 2744 consensus_queue.cc:237] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } }
I20260925 10:32:17.762020 2144 catalog_manager.cc:5495] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df reported cstate change: term changed from 0 to 1, leader changed from <none> to ce4182079abb4ae0bfd02ce5d1e141df (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4182079abb4ae0bfd02ce5d1e141df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 } health_report { overall_health: UNKNOWN } } }
I20260925 10:32:17.785492 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:17.779581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.0.97.1:41351)
0925 10:32:17.780106 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:17.780112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:17.780134 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:32:17.780230 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:17.780239 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:17.780304 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:17.780469 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:32:17.781272 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.782026 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.783412 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.783826 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.783949 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:17.784374 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:32:17.784913 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:32:17.785071 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:17.785129 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 10:32:17.785200 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":270,"threads_started":1}
I20260925 10:32:17.799688 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:17.794432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53909 (local address 127.0.97.2:44639)
0925 10:32:17.794589 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:17.794593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:17.795442 (+ 849us) server_negotiation.cc:407] Connection header received
0925 10:32:17.795580 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:17.795584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:17.795641 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:17.795711 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:32:17.796106 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.796873 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.798253 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.798702 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.798801 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:17.799161 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:32:17.799253 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:32:17.799316 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:17.799440 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 10:32:17.799493 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 10:32:17.799742 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:17.794256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44639 (local address 127.0.97.1:53909)
0925 10:32:17.794969 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:17.794984 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:17.795486 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:17.795736 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:17.795740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:17.795748 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:32:17.795988 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:17.795994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.796983 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.796986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:17.798133 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:17.798143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.798858 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.798862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:17.798982 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:17.799025 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:32:17.799333 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:32:17.799340 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:32:17.799399 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 10:32:17.799485 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":389,"threads_started":1}
I20260925 10:32:17.807142 2577 raft_consensus.cc:1261] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Refusing update from remote peer ce4182079abb4ae0bfd02ce5d1e141df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:17.807197 2444 raft_consensus.cc:1261] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce [term 1 FOLLOWER]: Refusing update from remote peer ce4182079abb4ae0bfd02ce5d1e141df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:17.807821 2757 consensus_queue.cc:1035] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [LEADER]: Connected to new peer: Peer: permanent_uuid: "98b2ae42644544c3ba0d44d87e02e76a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:17.808166 2744 consensus_queue.cc:1035] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0729ceedca9436d8691873a05be12ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:17.835727 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:17.828726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42389 (local address 127.0.97.1:41351)
0925 10:32:17.828908 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:17.828914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:17.830814 (+ 1900us) server_negotiation.cc:407] Connection header received
0925 10:32:17.831011 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:17.831018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:17.831103 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:17.831168 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:32:17.832079 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.832959 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.834144 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.834559 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.834645 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:17.835065 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:32:17.835159 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:32:17.835251 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:17.835392 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 10:32:17.835457 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:32:17.835850 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:17.828799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41351 (local address 127.0.97.2:42389)
0925 10:32:17.830683 (+ 1884us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:17.830698 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:17.830873 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:17.831571 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:17.831577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:17.831592 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:32:17.831954 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:17.831966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.833062 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.833065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:17.834031 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:17.834041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:17.834698 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:17.834702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:17.834861 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:17.834910 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:32:17.835261 (+ 351us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:32:17.835274 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 10:32:17.835373 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:32:17.835491 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1764,"thread_start_us":1644,"threads_started":1}
W20260925 10:32:17.836859 2711 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 10:32:19.421811 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:19.422021 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:19.425371 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdcc438de4d74eeaade5e4b60a337d7e: Sending DeleteTablet for 3 replicas of tablet f22e2ecd501546afbdba7532c09e6469
I20260925 10:32:19.426059 2279 tablet_service.cc:1514] Processing DeleteTablet for tablet f22e2ecd501546afbdba7532c09e6469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44522
I20260925 10:32:19.426065 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet f22e2ecd501546afbdba7532c09e6469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41774
I20260925 10:32:19.426084 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet f22e2ecd501546afbdba7532c09e6469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54714
I20260925 10:32:19.426735 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 2215
I20260925 10:32:19.429857 2794 tablet_replica.cc:330] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df: stopping tablet replica
I20260925 10:32:19.430151 2794 raft_consensus.cc:2229] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:32:19.430852 2794 raft_consensus.cc:2258] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:19.431713 2792 tablet_replica.cc:330] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce: stopping tablet replica
I20260925 10:32:19.432128 2794 ts_tablet_manager.cc:1905] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:19.432222 2793 tablet_replica.cc:330] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a: stopping tablet replica
I20260925 10:32:19.432406 2793 raft_consensus.cc:2229] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:19.432502 2792 raft_consensus.cc:2229] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:19.432683 2793 raft_consensus.cc:2258] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:19.432760 2792 raft_consensus.cc:2258] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:19.434005 2793 ts_tablet_manager.cc:1905] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:19.434118 2792 ts_tablet_manager.cc:1905] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:19.438663 2794 ts_tablet_manager.cc:1918] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:32:19.438802 2794 log.cc:1196] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/f22e2ecd501546afbdba7532c09e6469
I20260925 10:32:19.439193 2794 ts_tablet_manager.cc:1939] T f22e2ecd501546afbdba7532c09e6469 P ce4182079abb4ae0bfd02ce5d1e141df: Deleting consensus metadata
I20260925 10:32:19.440294 2131 catalog_manager.cc:4841] TS ce4182079abb4ae0bfd02ce5d1e141df (127.0.97.1:41351): tablet f22e2ecd501546afbdba7532c09e6469 (table test-table [id=b417cd0df94345b682fb99b371564738]) successfully deleted
I20260925 10:32:19.440420 2792 ts_tablet_manager.cc:1918] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:32:19.440420 2793 ts_tablet_manager.cc:1918] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:32:19.440542 2793 log.cc:1196] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/f22e2ecd501546afbdba7532c09e6469
I20260925 10:32:19.440585 2792 log.cc:1196] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/f22e2ecd501546afbdba7532c09e6469
I20260925 10:32:19.441007 2792 ts_tablet_manager.cc:1939] T f22e2ecd501546afbdba7532c09e6469 P b0729ceedca9436d8691873a05be12ce: Deleting consensus metadata
I20260925 10:32:19.441011 2793 ts_tablet_manager.cc:1939] T f22e2ecd501546afbdba7532c09e6469 P 98b2ae42644544c3ba0d44d87e02e76a: Deleting consensus metadata
I20260925 10:32:19.441970 2128 catalog_manager.cc:4841] TS b0729ceedca9436d8691873a05be12ce (127.0.97.2:44639): tablet f22e2ecd501546afbdba7532c09e6469 (table test-table [id=b417cd0df94345b682fb99b371564738]) successfully deleted
I20260925 10:32:19.442014 2131 catalog_manager.cc:4841] TS 98b2ae42644544c3ba0d44d87e02e76a (127.0.97.3:33975): tablet f22e2ecd501546afbdba7532c09e6469 (table test-table [id=b417cd0df94345b682fb99b371564738]) successfully deleted
Sep 25 10:32:19 dist-test-slave-dxdd krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:32:19.450201 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:19.427652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.0.97.1:41351)
0925 10:32:19.429363 (+ 1711us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:19.429371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:19.430614 (+ 1243us) server_negotiation.cc:407] Connection header received
0925 10:32:19.430686 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:19.430694 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:19.430836 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:19.430966 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:19.433227 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:19.434390 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:19.436189 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:19.436919 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:19.437019 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:19.445113 (+ 8094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:19.445132 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:19.445147 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:19.445178 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:19.447590 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:19.448213 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:19.448229 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:19.448233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:19.448353 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:19.448710 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:19.448715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:19.448718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:19.449033 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:19.449354 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:19.449643 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 10:32:19.449855 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1547,"thread_start_us":1407,"threads_started":1}
W20260925 10:32:19.450804 2345 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44548
W20260925 10:32:19.451334 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 2215: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:32:19.451514 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 2215
I20260925 10:32:19.473150 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 2354
Sep 25 10:32:19 dist-test-slave-dxdd krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:32:19.486596 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:19.473845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.0.97.2:44639)
0925 10:32:19.474350 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:19.474358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:19.474384 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:32:19.474432 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:19.474436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:19.474489 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:19.474574 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:19.475678 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:19.476353 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:19.477479 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:19.477902 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:19.477987 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:19.481786 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:19.481797 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:19.481811 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:19.481842 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:19.484385 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:19.484891 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:19.484898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:19.484900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:19.485003 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:19.485272 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:19.485275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:19.485277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:19.485577 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:19.485771 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:19.485985 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 10:32:19.486145 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":241,"threads_started":1}
W20260925 10:32:19.487197 2484 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41806
W20260925 10:32:19.487793 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 2354: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:32:19.487937 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 2354
I20260925 10:32:19.512290 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 2493
Sep 25 10:32:19 dist-test-slave-dxdd krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:32:19.525797 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:19.512950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.0.97.3:33975)
0925 10:32:19.513421 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:19.513425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:19.513441 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:32:19.513489 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:19.513494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:19.513568 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:19.513663 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:19.514979 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:19.515658 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:19.516767 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:19.517128 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:19.517215 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:19.521106 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:19.521119 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:19.521134 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:19.521166 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:19.523813 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:19.524347 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:19.524354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:19.524356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:19.524468 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:19.524756 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:19.524758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:19.524760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:19.525052 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:19.525252 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:19.525419 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 10:32:19.525554 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":268,"threads_started":1}
W20260925 10:32:19.526470 2623 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54722
W20260925 10:32:19.527041 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 2493: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:32:19.527199 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 2493
I20260925 10:32:19.545043 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 2112
I20260925 10:32:19.546137 2184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:19.659562 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 2112
2026-09-25T10:32:19Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8237 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:32:19 dist-test-slave-dxdd krb5kdc[2802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:32:19 dist-test-slave-dxdd krb5kdc[2802](info): set up 2 sockets
Sep 25 10:32:19 dist-test-slave-dxdd krb5kdc[2802](info): commencing operation
krb5kdc: starting...
W20260925 10:32:20.718353 388 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:32:20 dist-test-slave-dxdd krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:32:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:32:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:32:20.893085 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37181
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:21.200225 2815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:21.200515 2815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:21.200610 2815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:21.207609 2815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:21.207703 2815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:21.207748 2815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:21.207803 2815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:21.207839 2815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:32:21.217186 2815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:21.218422 2815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:21.219588 2815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:21.231196 2820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:21.231235 2821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:21.231251 2823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:21.231457 2815 server_base.cc:1034] running on GCE node
I20260925 10:32:21.232565 2815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:21.233896 2815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:21.235129 2815 hybrid_clock.cc:648] HybridClock initialized: now 1790332341235091 us; error 46 us; skew 500 ppm
I20260925 10:32:21.235471 2815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:21 dist-test-slave-dxdd krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:21.238396 2815 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:32:21.239713 2815 webserver.cc:460] Webserver started at http://127.0.97.62:33029/ using document root <none> and password file <none>
I20260925 10:32:21.240357 2815 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:21.240499 2815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:21.240803 2815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:21.243528 2815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "83e3892bcba4444d9ab65003d7970c8d"
format_stamp: "Formatted at 2026-09-25 10:32:21 on dist-test-slave-dxdd"
server_key: "bd04ccdfc774aced0c75ab4e40363751"
server_key_iv: "63052dc6946304fb39f1e19c477c1dca"
server_key_version: "encryptionkey@0"
I20260925 10:32:21.244267 2815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "83e3892bcba4444d9ab65003d7970c8d"
format_stamp: "Formatted at 2026-09-25 10:32:21 on dist-test-slave-dxdd"
server_key: "bd04ccdfc774aced0c75ab4e40363751"
server_key_iv: "63052dc6946304fb39f1e19c477c1dca"
server_key_version: "encryptionkey@0"
I20260925 10:32:21.248986 2815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 10:32:21.252096 2830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:21.253261 2815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:32:21.253429 2815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "83e3892bcba4444d9ab65003d7970c8d"
format_stamp: "Formatted at 2026-09-25 10:32:21 on dist-test-slave-dxdd"
server_key: "bd04ccdfc774aced0c75ab4e40363751"
server_key_iv: "63052dc6946304fb39f1e19c477c1dca"
server_key_version: "encryptionkey@0"
I20260925 10:32:21.253612 2815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:21.277370 2815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:21.281054 2815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:21.281328 2815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:21.298503 2815 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37181
I20260925 10:32:21.298533 2881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37181 every 8 connection(s)
I20260925 10:32:21.299783 2815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:32:21.303220 2882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:21.307859 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 2815
I20260925 10:32:21.308079 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:32:21.308678 388 external_mini_cluster.cc:1428] Setting key 972ee6f5ed5e86c7265f81646a1c1d7b
I20260925 10:32:21.314286 2882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d: Bootstrap starting.
I20260925 10:32:21.317962 2882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:21.319072 2882 log.cc:824] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:21.322000 2882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d: No bootstrap required, opened a new log
Sep 25 10:32:21 dist-test-slave-dxdd krb5kdc[2802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:21.331348 2882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83e3892bcba4444d9ab65003d7970c8d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37181 } }
I20260925 10:32:21.331764 2882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:21.331868 2882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e3892bcba4444d9ab65003d7970c8d, State: Initialized, Role: FOLLOWER
I20260925 10:32:21.332456 2882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83e3892bcba4444d9ab65003d7970c8d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37181 } }
I20260925 10:32:21.332747 2885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:21.310571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42770 (local address 127.0.97.62:37181)
0925 10:32:21.311355 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:21.311374 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:21.311423 (+ 49us) server_negotiation.cc:407] Connection header received
0925 10:32:21.313681 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:21.313705 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:21.314146 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:21.316389 (+ 2243us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:21.316844 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:21.318713 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:21.320266 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:21.321211 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:21.321381 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:21.324980 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:21.324989 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:21.325010 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:21.325048 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:21.328776 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:21.329396 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:21.329404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:21.329411 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:21.329599 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:21.329898 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:21.329904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:21.329913 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:21.330585 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:21.330836 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:21.331141 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 10:32:21.331520 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":316,"threads_started":1}
I20260925 10:32:21.332746 2882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:32:21.332913 2882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:32:21.333094 2882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:21.335774 2882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83e3892bcba4444d9ab65003d7970c8d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37181 } }
I20260925 10:32:21.336140 2882 leader_election.cc:302] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83e3892bcba4444d9ab65003d7970c8d; no voters:
I20260925 10:32:21.336566 2882 leader_election.cc:288] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:32:21.336721 2887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:21.337415 2887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [term 1 LEADER]: Becoming Leader. State: Replica: 83e3892bcba4444d9ab65003d7970c8d, State: Running, Role: LEADER
I20260925 10:32:21.337888 2887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83e3892bcba4444d9ab65003d7970c8d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37181 } }
I20260925 10:32:21.338546 2882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:21.340616 2888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83e3892bcba4444d9ab65003d7970c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83e3892bcba4444d9ab65003d7970c8d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37181 } } }
I20260925 10:32:21.340893 2889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83e3892bcba4444d9ab65003d7970c8d. Latest consensus state: current_term: 1 leader_uuid: "83e3892bcba4444d9ab65003d7970c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83e3892bcba4444d9ab65003d7970c8d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37181 } } }
I20260925 10:32:21.341117 2888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:21.341197 2889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:21.341919 2896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:21.348776 2896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:21.356524 2896 catalog_manager.cc:1347] Generated new cluster ID: 18be8b65aa774e62af3090f121c57eb9
I20260925 10:32:21.356639 2896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:21.375291 2896 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:32:21.377036 2896 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:21.383239 2896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83e3892bcba4444d9ab65003d7970c8d: Generated new TSK 0
I20260925 10:32:21.383961 2896 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:32:21.438462 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37181
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:21.767233 2908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:21.767474 2908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:21.767679 2908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:21.775092 2908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:21.775211 2908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:21.775396 2908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:32:21.784679 2908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:21.785992 2908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:21.787428 2908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:21.799706 2914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:21.799865 2913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:21.799862 2916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:21.800763 2908 server_base.cc:1034] running on GCE node
I20260925 10:32:21.801286 2908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:21.802138 2908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:21.803330 2908 hybrid_clock.cc:648] HybridClock initialized: now 1790332341803267 us; error 51 us; skew 500 ppm
I20260925 10:32:21.803574 2908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:21 dist-test-slave-dxdd krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:21.806568 2908 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:32:21.807834 2908 webserver.cc:460] Webserver started at http://127.0.97.1:42529/ using document root <none> and password file <none>
I20260925 10:32:21.808424 2908 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:21.808519 2908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:21.808758 2908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:21.811520 2908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "9063a1a45d934667a434ff554a672e25"
format_stamp: "Formatted at 2026-09-25 10:32:21 on dist-test-slave-dxdd"
server_key: "bf4e4319c5d975fed2f57dca0f4a8869"
server_key_iv: "8596f2f058da0c129f9f252b33751a5a"
server_key_version: "encryptionkey@0"
I20260925 10:32:21.812189 2908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9063a1a45d934667a434ff554a672e25"
format_stamp: "Formatted at 2026-09-25 10:32:21 on dist-test-slave-dxdd"
server_key: "bf4e4319c5d975fed2f57dca0f4a8869"
server_key_iv: "8596f2f058da0c129f9f252b33751a5a"
server_key_version: "encryptionkey@0"
I20260925 10:32:21.817019 2908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:32:21.820067 2923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:21.821285 2908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 10:32:21.821442 2908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "9063a1a45d934667a434ff554a672e25"
format_stamp: "Formatted at 2026-09-25 10:32:21 on dist-test-slave-dxdd"
server_key: "bf4e4319c5d975fed2f57dca0f4a8869"
server_key_iv: "8596f2f058da0c129f9f252b33751a5a"
server_key_version: "encryptionkey@0"
I20260925 10:32:21.821635 2908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:21.836443 2908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:21.840380 2908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:21.840678 2908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:21.841568 2908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:21.842859 2908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:21.842926 2908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:21.843087 2908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:21.843155 2908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:21.875392 2908 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45883
I20260925 10:32:21.875447 3035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45883 every 8 connection(s)
I20260925 10:32:21.876621 2908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:32:21.886044 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 2908
I20260925 10:32:21.886253 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:32:21.886986 388 external_mini_cluster.cc:1428] Setting key 95646933eff35fd4f8df57e02560a243
Sep 25 10:32:21 dist-test-slave-dxdd krb5kdc[2802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:21.897457 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:21.879097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49901 (local address 127.0.97.62:37181)
0925 10:32:21.879526 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:21.879532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:21.880554 (+ 1022us) server_negotiation.cc:407] Connection header received
0925 10:32:21.882327 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:21.882352 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:21.882440 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:21.882590 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:21.884311 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:21.885017 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:21.886603 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:21.887251 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:21.887374 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:21.891008 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:21.891018 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:21.891026 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:21.891070 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:21.894473 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:21.895380 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:21.895383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:21.895386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:21.895502 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:21.895971 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:21.895974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:21.895976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:21.896195 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:21.896299 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:21.897086 (+ 787us) server_negotiation.cc:299] Negotiation successful
0925 10:32:21.897225 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":212,"threads_started":1}
I20260925 10:32:21.898958 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:21.879692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37181 (local address 127.0.97.1:49901)
0925 10:32:21.880368 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:21.880421 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:21.882021 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:21.882819 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:21.882826 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:21.883309 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:21.884136 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:21.884158 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:21.885175 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:21.885182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:21.886476 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:21.886487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:21.887458 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:21.887463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:21.887663 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:21.888382 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:21.888403 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:21.890797 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:21.894623 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:21.894635 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:21.894653 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:21.895262 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:21.895611 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:21.895617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:21.895622 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:21.895870 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:21.896310 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:21.896320 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:21.896743 (+ 423us) client_negotiation.cc:769] Sending connection context
0925 10:32:21.897019 (+ 276us) client_negotiation.cc:240] Negotiation successful
0925 10:32:21.897546 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":270,"threads_started":1}
I20260925 10:32:21.900444 3036 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37181
I20260925 10:32:21.900739 3036 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:21.901448 3036 heartbeater.cc:511] Master 127.0.97.62:37181 requested a full tablet report, sending...
I20260925 10:32:21.903332 2847 ts_manager.cc:194] Registered new tserver with Master: 9063a1a45d934667a434ff554a672e25 (127.0.97.1:45883)
I20260925 10:32:21.904693 2847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49901
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:32:21.927392 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37181
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:22.239914 3042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:22.240142 3042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:22.240226 3042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:22.247056 3042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:22.247140 3042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:22.247238 3042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:32:22.256754 3042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:22.257949 3042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:22.259320 3042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:22.271822 3050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:22.271867 3047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:22.271906 3048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:22.272436 3042 server_base.cc:1034] running on GCE node
I20260925 10:32:22.272965 3042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:22.273759 3042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:22.274964 3042 hybrid_clock.cc:648] HybridClock initialized: now 1790332342274930 us; error 46 us; skew 500 ppm
I20260925 10:32:22.275240 3042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:22 dist-test-slave-dxdd krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:22.277944 3042 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:32:22.279301 3042 webserver.cc:460] Webserver started at http://127.0.97.2:35355/ using document root <none> and password file <none>
I20260925 10:32:22.279922 3042 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:22.280055 3042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:22.280339 3042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:22.283094 3042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "53a55613ae704b3c914e2cb345a851f2"
format_stamp: "Formatted at 2026-09-25 10:32:22 on dist-test-slave-dxdd"
server_key: "de909fe14fe1f780a41bb85b0fd6e1f2"
server_key_iv: "5b69d67494526d1144e5f2736bd6c72d"
server_key_version: "encryptionkey@0"
I20260925 10:32:22.283830 3042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "53a55613ae704b3c914e2cb345a851f2"
format_stamp: "Formatted at 2026-09-25 10:32:22 on dist-test-slave-dxdd"
server_key: "de909fe14fe1f780a41bb85b0fd6e1f2"
server_key_iv: "5b69d67494526d1144e5f2736bd6c72d"
server_key_version: "encryptionkey@0"
I20260925 10:32:22.288442 3042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 10:32:22.291508 3057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:22.292613 3042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:32:22.292783 3042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "53a55613ae704b3c914e2cb345a851f2"
format_stamp: "Formatted at 2026-09-25 10:32:22 on dist-test-slave-dxdd"
server_key: "de909fe14fe1f780a41bb85b0fd6e1f2"
server_key_iv: "5b69d67494526d1144e5f2736bd6c72d"
server_key_version: "encryptionkey@0"
I20260925 10:32:22.292963 3042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:22.311673 3042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:22.315155 3042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:22.315428 3042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:22.316254 3042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:22.317574 3042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:22.317672 3042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:22.317806 3042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:22.317898 3042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:22.348850 3042 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37883
I20260925 10:32:22.348892 3169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37883 every 8 connection(s)
I20260925 10:32:22.350052 3042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:32:22.352449 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 3042
I20260925 10:32:22.352641 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:32:22.353068 388 external_mini_cluster.cc:1428] Setting key f4bab5cb65cbddaa8e31927125fccbd8
Sep 25 10:32:22 dist-test-slave-dxdd krb5kdc[2802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:22.369732 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:22.352468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58103 (local address 127.0.97.62:37181)
0925 10:32:22.352656 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:22.352663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:22.353961 (+ 1298us) server_negotiation.cc:407] Connection header received
0925 10:32:22.355807 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:22.355812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:22.355891 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:22.356002 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:22.357563 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:22.358296 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:22.359832 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:22.360221 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:22.360356 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:22.364109 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:22.364114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:22.364117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:22.364145 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:22.366704 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:22.367596 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:22.367599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:22.367601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:22.367721 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:22.368179 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:22.368182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:22.368184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:22.368433 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:22.368534 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:22.369350 (+ 816us) server_negotiation.cc:299] Negotiation successful
0925 10:32:22.369496 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260925 10:32:22.371023 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:22.352989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37181 (local address 127.0.97.2:58103)
0925 10:32:22.353800 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:22.353856 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:22.355546 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:22.356228 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:22.356237 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:22.356817 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:22.357396 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:22.357414 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:22.358481 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:22.358485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:22.359659 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:22.359670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:22.360438 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:22.360445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:22.360658 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:22.361322 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:22.361348 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:22.363906 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:22.366835 (+ 2929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:22.366855 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:22.366873 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:22.367489 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:22.367828 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:22.367835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:22.367840 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:22.368069 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:22.368557 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:22.368565 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:22.368985 (+ 420us) client_negotiation.cc:769] Sending connection context
0925 10:32:22.369306 (+ 321us) client_negotiation.cc:240] Negotiation successful
0925 10:32:22.369808 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":365,"threads_started":1}
I20260925 10:32:22.372422 3170 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260925 10:32:22.372694 3170 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:22.373543 3170 heartbeater.cc:511] Master 127.0.97.62:37181 requested a full tablet report, sending...
I20260925 10:32:22.375025 2847 ts_manager.cc:194] Registered new tserver with Master: 53a55613ae704b3c914e2cb345a851f2 (127.0.97.2:37883)
I20260925 10:32:22.375793 2847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:32:22.392617 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37181
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:22.695379 3175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:22.695641 3175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:22.695725 3175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:22.702392 3175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:22.702479 3175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:22.702585 3175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:32:22.711432 3175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:22.712491 3175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:22.713649 3175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:22.725978 3180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:22.726032 3181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:22.726054 3183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:22.726866 3175 server_base.cc:1034] running on GCE node
I20260925 10:32:22.727388 3175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:22.728227 3175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:22.729434 3175 hybrid_clock.cc:648] HybridClock initialized: now 1790332342729397 us; error 49 us; skew 500 ppm
I20260925 10:32:22.729709 3175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:22 dist-test-slave-dxdd krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:22.732524 3175 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:32:22.733836 3175 webserver.cc:460] Webserver started at http://127.0.97.3:39715/ using document root <none> and password file <none>
I20260925 10:32:22.734489 3175 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:22.734622 3175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:22.734905 3175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:22.737573 3175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "f628c5d215f940fab106954e06ffc582"
format_stamp: "Formatted at 2026-09-25 10:32:22 on dist-test-slave-dxdd"
server_key: "2f16967081492a85c85076bb01307b45"
server_key_iv: "3d04a50ef373f8eefb84d0b1a5e60087"
server_key_version: "encryptionkey@0"
I20260925 10:32:22.738273 3175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f628c5d215f940fab106954e06ffc582"
format_stamp: "Formatted at 2026-09-25 10:32:22 on dist-test-slave-dxdd"
server_key: "2f16967081492a85c85076bb01307b45"
server_key_iv: "3d04a50ef373f8eefb84d0b1a5e60087"
server_key_version: "encryptionkey@0"
I20260925 10:32:22.742858 3175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:32:22.745764 3190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:22.746976 3175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:32:22.747136 3175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "f628c5d215f940fab106954e06ffc582"
format_stamp: "Formatted at 2026-09-25 10:32:22 on dist-test-slave-dxdd"
server_key: "2f16967081492a85c85076bb01307b45"
server_key_iv: "3d04a50ef373f8eefb84d0b1a5e60087"
server_key_version: "encryptionkey@0"
I20260925 10:32:22.747313 3175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:22.768465 3175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:22.772123 3175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:22.772431 3175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:22.773283 3175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:22.774488 3175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:22.774588 3175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:22.774739 3175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:22.774806 3175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:22.806249 3175 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42329
I20260925 10:32:22.806356 3302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42329 every 8 connection(s)
I20260925 10:32:22.807474 3175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:32:22.817272 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 3175
I20260925 10:32:22.817476 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:32:22.817906 388 external_mini_cluster.cc:1428] Setting key 053cbc5aab6300afe27a5c912b1a516f
Sep 25 10:32:22 dist-test-slave-dxdd krb5kdc[2802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:22.825837 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:22.809807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39717 (local address 127.0.97.62:37181)
0925 10:32:22.809969 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:22.809973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:22.811116 (+ 1143us) server_negotiation.cc:407] Connection header received
0925 10:32:22.812765 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:22.812769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:22.812837 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:22.812922 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:22.814263 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:22.815003 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:22.816400 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:22.816775 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:22.816870 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:22.820523 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:22.820528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:22.820531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:22.820557 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:22.823032 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:22.823688 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:22.823695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:22.823698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:22.823811 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:22.824206 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:22.824213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:22.824217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:22.824524 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:22.824666 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:22.825274 (+ 608us) server_negotiation.cc:299] Negotiation successful
0925 10:32:22.825494 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:32:22.826828 3305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:22.810213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37181 (local address 127.0.97.3:39717)
0925 10:32:22.810958 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:22.810998 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:22.812540 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:22.813113 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:22.813120 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:22.813577 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:22.814112 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:22.814127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:22.815125 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:22.815129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:22.816288 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:22.816298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:22.816962 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:22.816967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:22.817141 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:22.817884 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:22.817913 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:22.820319 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:22.823170 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:22.823178 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:22.823191 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:22.823574 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:22.823927 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:22.823931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:22.823935 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:22.824101 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:22.824643 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:22.824650 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:22.824941 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 10:32:22.825181 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 10:32:22.825721 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":289,"threads_started":1}
I20260925 10:32:22.828058 3303 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37181
I20260925 10:32:22.828326 3303 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:22.828943 3303 heartbeater.cc:511] Master 127.0.97.62:37181 requested a full tablet report, sending...
I20260925 10:32:22.830176 2847 ts_manager.cc:194] Registered new tserver with Master: f628c5d215f940fab106954e06ffc582 (127.0.97.3:42329)
I20260925 10:32:22.831064 2847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39717
I20260925 10:32:22.832715 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 10:32:22.860164 3038 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:32:22.857956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.0.97.62:37181)
0925 10:32:22.858150 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:22.858157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:22.858628 (+ 471us) server_negotiation.cc:407] Connection header received
0925 10:32:22.858746 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:22.858750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:22.858808 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:22.858888 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:22.859900 (+ 1012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52130: BlockingRecv error: recv got EOF from 127.0.0.1:52130 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:32:22.863186 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 2908
W20260925 10:32:22.865724 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 2908: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:45883: server requires authentication, but client does not have Kerberos credentials available
I20260925 10:32:22.865868 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 2908
I20260925 10:32:22.880875 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 3042
W20260925 10:32:22.883020 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 3042: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:37883: server requires authentication, but client does not have Kerberos credentials available
I20260925 10:32:22.883163 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 3042
I20260925 10:32:22.897425 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 3175
W20260925 10:32:22.899978 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 3175: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:42329: server requires authentication, but client does not have Kerberos credentials available
I20260925 10:32:22.900102 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 3175
I20260925 10:32:22.914178 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 2815
I20260925 10:32:22.915208 2877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:23.013085 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 2815
2026-09-25T10:32:23Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3352 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:32:23 dist-test-slave-dxdd krb5kdc[3320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:32:23 dist-test-slave-dxdd krb5kdc[3320](info): set up 2 sockets
Sep 25 10:32:23 dist-test-slave-dxdd krb5kdc[3320](info): commencing operation
krb5kdc: starting...
W20260925 10:32:24.070573 388 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:32:24 dist-test-slave-dxdd krb5kdc[3320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:32:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:32:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:32:24.243739 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36055
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:24.543854 3333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:24.544106 3333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:24.544267 3333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:24.551360 3333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:24.551479 3333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:24.551558 3333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:24.551664 3333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:24.551751 3333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:32:24.561221 3333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:24.562422 3333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:24.563720 3333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:24.574977 3339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:24.575071 3341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:24.575039 3338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:24.576058 3333 server_base.cc:1034] running on GCE node
I20260925 10:32:24.576699 3333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:24.577952 3333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:24.579150 3333 hybrid_clock.cc:648] HybridClock initialized: now 1790332344579105 us; error 30 us; skew 500 ppm
I20260925 10:32:24.579468 3333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:24 dist-test-slave-dxdd krb5kdc[3320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:24.587316 3333 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:32:24.588649 3333 webserver.cc:460] Webserver started at http://127.0.97.62:33773/ using document root <none> and password file <none>
I20260925 10:32:24.589241 3333 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:24.589332 3333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:24.589558 3333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:24.592280 3333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "c88409028f134264b9ea5e0080a48f3a"
format_stamp: "Formatted at 2026-09-25 10:32:24 on dist-test-slave-dxdd"
server_key: "7a4c84080ef177765b8234116845ce9d"
server_key_iv: "8665f06127a264e2a8ed477413f9f560"
server_key_version: "encryptionkey@0"
I20260925 10:32:24.592981 3333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "c88409028f134264b9ea5e0080a48f3a"
format_stamp: "Formatted at 2026-09-25 10:32:24 on dist-test-slave-dxdd"
server_key: "7a4c84080ef177765b8234116845ce9d"
server_key_iv: "8665f06127a264e2a8ed477413f9f560"
server_key_version: "encryptionkey@0"
I20260925 10:32:24.597585 3333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 10:32:24.600819 3348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:24.601953 3333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:32:24.602123 3333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "c88409028f134264b9ea5e0080a48f3a"
format_stamp: "Formatted at 2026-09-25 10:32:24 on dist-test-slave-dxdd"
server_key: "7a4c84080ef177765b8234116845ce9d"
server_key_iv: "8665f06127a264e2a8ed477413f9f560"
server_key_version: "encryptionkey@0"
I20260925 10:32:24.602313 3333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:24.629416 3333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:24.633124 3333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:24.633409 3333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:24.650979 3333 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36055
I20260925 10:32:24.651017 3399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36055 every 8 connection(s)
I20260925 10:32:24.652261 3333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:32:24.655627 3400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:24.659303 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 3333
I20260925 10:32:24.659494 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:32:24.660226 388 external_mini_cluster.cc:1428] Setting key 5066ae2224db5d5c71a81e3b426fe4b7
I20260925 10:32:24.668292 3400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a: Bootstrap starting.
I20260925 10:32:24.671885 3400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:24.673147 3400 log.cc:824] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:32:24 dist-test-slave-dxdd krb5kdc[3320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:24.676118 3400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a: No bootstrap required, opened a new log
I20260925 10:32:24.682992 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:24.662052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.0.97.62:36055)
0925 10:32:24.662835 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:24.662845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:24.662883 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:32:24.664404 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:24.664427 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:24.664794 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:24.665106 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:24.666780 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:24.668059 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:24.669774 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:24.670479 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:24.670596 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:24.675171 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:24.675183 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:24.675201 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:24.675245 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:24.678997 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:24.679571 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:24.679579 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:24.679587 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:24.679783 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:24.680095 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:24.680102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:24.680110 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:24.680737 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:24.681011 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:24.681332 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 10:32:24.681736 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":387,"threads_started":1}
I20260925 10:32:24.686595 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:32:24.687282 3400 raft_consensus.cc:348] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c88409028f134264b9ea5e0080a48f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36055 } }
I20260925 10:32:24.687592 3400 raft_consensus.cc:374] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:24.687698 3400 raft_consensus.cc:729] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c88409028f134264b9ea5e0080a48f3a, State: Initialized, Role: FOLLOWER
I20260925 10:32:24.688223 3400 consensus_queue.cc:260] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c88409028f134264b9ea5e0080a48f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36055 } }
I20260925 10:32:24.688443 3400 raft_consensus.cc:388] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:32:24.688580 3400 raft_consensus.cc:482] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:32:24.688761 3400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:24.690786 3400 raft_consensus.cc:504] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c88409028f134264b9ea5e0080a48f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36055 } }
I20260925 10:32:24.691229 3400 leader_election.cc:302] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c88409028f134264b9ea5e0080a48f3a; no voters:
I20260925 10:32:24.691793 3400 leader_election.cc:288] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:32:24.691919 3405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:24.693593 3400 sys_catalog.cc:564] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:24.694615 3405 raft_consensus.cc:686] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [term 1 LEADER]: Becoming Leader. State: Replica: c88409028f134264b9ea5e0080a48f3a, State: Running, Role: LEADER
I20260925 10:32:24.695233 3405 consensus_queue.cc:237] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c88409028f134264b9ea5e0080a48f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36055 } }
I20260925 10:32:24.698203 3406 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c88409028f134264b9ea5e0080a48f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c88409028f134264b9ea5e0080a48f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36055 } } }
I20260925 10:32:24.699541 3406 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:24.698560 3408 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c88409028f134264b9ea5e0080a48f3a. Latest consensus state: current_term: 1 leader_uuid: "c88409028f134264b9ea5e0080a48f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c88409028f134264b9ea5e0080a48f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36055 } } }
I20260925 10:32:24.701153 3408 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:24.703830 3418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:24.708630 3418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:24.717280 3418 catalog_manager.cc:1347] Generated new cluster ID: f12d856a14ea45a8a36433339336cf9b
I20260925 10:32:24.717429 3418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:24.726190 3418 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:32:24.727851 3418 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:24.728781 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:24.719747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.0.97.62:36055)
0925 10:32:24.719919 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:24.719926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:24.720464 (+ 538us) server_negotiation.cc:407] Connection header received
0925 10:32:24.720656 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:24.720661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:24.720732 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:24.720824 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:24.722099 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:24.723302 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:24.725101 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:24.725710 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:24.725829 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:24.727040 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:24.727046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:24.727050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:24.727796 (+ 746us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:24.727898 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:24.727934 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:24.728065 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:24.728271 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 10:32:24.728398 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:32:24.738334 3418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c88409028f134264b9ea5e0080a48f3a: Generated new TSK 0
I20260925 10:32:24.739161 3418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:32:24.748129 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 3333
I20260925 10:32:24.749468 3395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:24.834609 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 3333
2026-09-25T10:32:24Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1814 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:32:24 dist-test-slave-dxdd krb5kdc[3434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:32:24 dist-test-slave-dxdd krb5kdc[3434](info): set up 2 sockets
Sep 25 10:32:24 dist-test-slave-dxdd krb5kdc[3434](info): commencing operation
krb5kdc: starting...
W20260925 10:32:25.894557 388 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:32:26 dist-test-slave-dxdd krb5kdc[3434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:32:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:32:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:32:26.079114 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42649
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:26.383332 3447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:26.383587 3447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:26.383730 3447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:26.390654 3447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:26.390741 3447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:26.390837 3447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:26.390929 3447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:26.390990 3447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:32:26.400264 3447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:26.401451 3447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:26.402714 3447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:26.413635 3453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:26.413681 3452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:26.413645 3455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:26.414544 3447 server_base.cc:1034] running on GCE node
I20260925 10:32:26.415241 3447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:26.416479 3447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:26.417665 3447 hybrid_clock.cc:648] HybridClock initialized: now 1790332346417623 us; error 35 us; skew 500 ppm
I20260925 10:32:26.417940 3447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:26 dist-test-slave-dxdd krb5kdc[3434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:26.420823 3447 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:32:26.422047 3447 webserver.cc:460] Webserver started at http://127.0.97.62:39301/ using document root <none> and password file <none>
I20260925 10:32:26.422713 3447 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:26.422808 3447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:26.423040 3447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:26.425751 3447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "6893a29d4da8478cb73e409a3e4b099b"
format_stamp: "Formatted at 2026-09-25 10:32:26 on dist-test-slave-dxdd"
server_key: "f9cbcbcbb20b3a19b6e0b2b6ab3731a4"
server_key_iv: "76c9b5f9982781a07eb3d6bd92aff189"
server_key_version: "encryptionkey@0"
I20260925 10:32:26.426553 3447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "6893a29d4da8478cb73e409a3e4b099b"
format_stamp: "Formatted at 2026-09-25 10:32:26 on dist-test-slave-dxdd"
server_key: "f9cbcbcbb20b3a19b6e0b2b6ab3731a4"
server_key_iv: "76c9b5f9982781a07eb3d6bd92aff189"
server_key_version: "encryptionkey@0"
I20260925 10:32:26.431186 3447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:32:26.434217 3462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:26.435424 3447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 10:32:26.435593 3447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "6893a29d4da8478cb73e409a3e4b099b"
format_stamp: "Formatted at 2026-09-25 10:32:26 on dist-test-slave-dxdd"
server_key: "f9cbcbcbb20b3a19b6e0b2b6ab3731a4"
server_key_iv: "76c9b5f9982781a07eb3d6bd92aff189"
server_key_version: "encryptionkey@0"
I20260925 10:32:26.435762 3447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:26.455647 3447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:26.459582 3447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:26.459849 3447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:26.476732 3447 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42649
I20260925 10:32:26.476769 3513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42649 every 8 connection(s)
I20260925 10:32:26.477962 3447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:32:26.481341 3514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:26.485536 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 3447
I20260925 10:32:26.485730 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:32:26.486347 388 external_mini_cluster.cc:1428] Setting key d3e1e1e1982110339cca989c811d1b8e
I20260925 10:32:26.491102 3514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b: Bootstrap starting.
I20260925 10:32:26.493852 3514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:26.494825 3514 log.cc:824] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:26.497419 3514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b: No bootstrap required, opened a new log
Sep 25 10:32:26 dist-test-slave-dxdd krb5kdc[3434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:26.504612 3514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893a29d4da8478cb73e409a3e4b099b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42649 } }
I20260925 10:32:26.504967 3514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:26.505100 3514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6893a29d4da8478cb73e409a3e4b099b, State: Initialized, Role: FOLLOWER
I20260925 10:32:26.505718 3514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893a29d4da8478cb73e409a3e4b099b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42649 } }
I20260925 10:32:26.505954 3514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:32:26.506103 3514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:32:26.506353 3514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:26.507407 3517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:26.488161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45424 (local address 127.0.97.62:42649)
0925 10:32:26.488998 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:26.489008 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:26.489048 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:32:26.490653 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:26.490673 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:26.491006 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:26.491383 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:26.492672 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:26.493878 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:26.495158 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:26.496266 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:26.496479 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:26.500279 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:26.500290 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:26.500302 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:26.500340 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:26.503445 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:26.504107 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:26.504122 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:26.504127 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:26.504275 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:26.504632 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:26.504638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:26.504641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:26.505252 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:26.505589 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:26.505957 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 10:32:26.506404 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":388,"threads_started":1}
I20260925 10:32:26.508761 3514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893a29d4da8478cb73e409a3e4b099b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42649 } }
I20260925 10:32:26.509281 3514 leader_election.cc:302] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6893a29d4da8478cb73e409a3e4b099b; no voters:
I20260925 10:32:26.509858 3514 leader_election.cc:288] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:32:26.509968 3519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:26.510735 3519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [term 1 LEADER]: Becoming Leader. State: Replica: 6893a29d4da8478cb73e409a3e4b099b, State: Running, Role: LEADER
I20260925 10:32:26.511288 3519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893a29d4da8478cb73e409a3e4b099b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42649 } }
I20260925 10:32:26.511966 3514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:26.514081 3521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6893a29d4da8478cb73e409a3e4b099b. Latest consensus state: current_term: 1 leader_uuid: "6893a29d4da8478cb73e409a3e4b099b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893a29d4da8478cb73e409a3e4b099b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42649 } } }
I20260925 10:32:26.514503 3521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:26.514555 3520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6893a29d4da8478cb73e409a3e4b099b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893a29d4da8478cb73e409a3e4b099b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42649 } } }
I20260925 10:32:26.514900 3520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:26.517453 3527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:26.522373 3527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:26.530512 3527 catalog_manager.cc:1347] Generated new cluster ID: a3278488105449658cdf24ab4b95f38b
I20260925 10:32:26.530606 3527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:26.545537 3527 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:32:26.546741 3527 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:26.553049 3527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6893a29d4da8478cb73e409a3e4b099b: Generated new TSK 0
I20260925 10:32:26.553689 3527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:32:26.605304 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42649
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:26.912048 3540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:26.912402 3540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:26.912566 3540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:26.919857 3540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:26.919973 3540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:26.920149 3540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:32:26.929258 3540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:26.930492 3540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:26.931743 3540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:26.944711 3545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:26.944624 3548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:26.944741 3546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:26.945363 3540 server_base.cc:1034] running on GCE node
I20260925 10:32:26.945899 3540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:26.946768 3540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:26.947952 3540 hybrid_clock.cc:648] HybridClock initialized: now 1790332346947901 us; error 43 us; skew 500 ppm
I20260925 10:32:26.948228 3540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:26 dist-test-slave-dxdd krb5kdc[3434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:26.951103 3540 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:32:26.952394 3540 webserver.cc:460] Webserver started at http://127.0.97.1:37679/ using document root <none> and password file <none>
I20260925 10:32:26.953044 3540 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:26.953176 3540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:26.953445 3540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:26.956084 3540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "7b181ba0905947b9a504007dff3d4396"
format_stamp: "Formatted at 2026-09-25 10:32:26 on dist-test-slave-dxdd"
server_key: "562b00707d16b3c99bfb8de19b4aa371"
server_key_iv: "01e8f9563d0938448b637ce0a93fb292"
server_key_version: "encryptionkey@0"
I20260925 10:32:26.956812 3540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7b181ba0905947b9a504007dff3d4396"
format_stamp: "Formatted at 2026-09-25 10:32:26 on dist-test-slave-dxdd"
server_key: "562b00707d16b3c99bfb8de19b4aa371"
server_key_iv: "01e8f9563d0938448b637ce0a93fb292"
server_key_version: "encryptionkey@0"
I20260925 10:32:26.961251 3540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 10:32:26.964432 3555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:26.965538 3540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:32:26.965718 3540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "7b181ba0905947b9a504007dff3d4396"
format_stamp: "Formatted at 2026-09-25 10:32:26 on dist-test-slave-dxdd"
server_key: "562b00707d16b3c99bfb8de19b4aa371"
server_key_iv: "01e8f9563d0938448b637ce0a93fb292"
server_key_version: "encryptionkey@0"
I20260925 10:32:26.965909 3540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:26.976953 3540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:26.980432 3540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:26.980708 3540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:26.981537 3540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:26.982779 3540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:26.982880 3540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:26.983016 3540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:26.983085 3540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:27.014494 3540 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35043
I20260925 10:32:27.014539 3667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35043 every 8 connection(s)
I20260925 10:32:27.015717 3540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:32:27.020846 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 3540
I20260925 10:32:27.021057 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:32:27.021653 388 external_mini_cluster.cc:1428] Setting key 7c012a5a573c99e3b1d1a7cbb160895b
Sep 25 10:32:27 dist-test-slave-dxdd krb5kdc[3434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:27.035981 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:27.018217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45049 (local address 127.0.97.62:42649)
0925 10:32:27.018684 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:27.018689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:27.019653 (+ 964us) server_negotiation.cc:407] Connection header received
0925 10:32:27.021453 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:27.021458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:27.021554 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:27.021669 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:27.023456 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:27.024495 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:27.025953 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:27.026674 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:27.026814 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:27.030492 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:27.030498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:27.030502 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:27.030541 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:27.033007 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:27.033672 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:27.033677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:27.033680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:27.033797 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:27.034194 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:27.034199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:27.034203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:27.034515 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:27.034656 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:27.035436 (+ 780us) server_negotiation.cc:299] Negotiation successful
0925 10:32:27.035638 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":258,"threads_started":1}
I20260925 10:32:27.036873 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:27.018817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42649 (local address 127.0.97.1:45049)
0925 10:32:27.019497 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:27.019552 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:27.021197 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:27.021887 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:27.021895 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:27.022522 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:27.023268 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:27.023287 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:27.024648 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:27.024652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:27.025814 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:27.025828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:27.026859 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:27.026863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:27.027207 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:27.027835 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:27.027855 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:27.030256 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:27.033142 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:27.033152 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:27.033166 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:27.033553 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:27.033905 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:27.033910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:27.033913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:27.034082 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:27.034631 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:27.034638 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:27.035058 (+ 420us) client_negotiation.cc:769] Sending connection context
0925 10:32:27.035338 (+ 280us) client_negotiation.cc:240] Negotiation successful
0925 10:32:27.035816 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":281,"threads_started":1}
I20260925 10:32:27.038376 3668 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42649
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260925 10:32:27.038666 3668 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:27.039333 3668 heartbeater.cc:511] Master 127.0.97.62:42649 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260925 10:32:27.041431 3479 ts_manager.cc:194] Registered new tserver with Master: 7b181ba0905947b9a504007dff3d4396 (127.0.97.1:35043)
I20260925 10:32:27.042871 3479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:32:27.063433 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42649
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:27.364537 3674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:27.364781 3674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:27.364869 3674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:27.372071 3674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:27.372190 3674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:27.372375 3674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:32:27.381505 3674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:27.382656 3674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:27.383867 3674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:27.396078 3679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:27.396116 3680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:27.396167 3682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:27.396903 3674 server_base.cc:1034] running on GCE node
I20260925 10:32:27.397418 3674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:27.398379 3674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:27.399627 3674 hybrid_clock.cc:648] HybridClock initialized: now 1790332347399573 us; error 46 us; skew 500 ppm
I20260925 10:32:27.399915 3674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:27 dist-test-slave-dxdd krb5kdc[3434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:27.402890 3674 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:32:27.404187 3674 webserver.cc:460] Webserver started at http://127.0.97.2:43701/ using document root <none> and password file <none>
I20260925 10:32:27.404785 3674 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:27.404877 3674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:27.405114 3674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:27.407836 3674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "0182845d45974fe8b8a5097258982ed3"
format_stamp: "Formatted at 2026-09-25 10:32:27 on dist-test-slave-dxdd"
server_key: "ca695bb899e2a6ceeb3577297211d5c6"
server_key_iv: "e3e2c2fe40af7c21944d43311b09398f"
server_key_version: "encryptionkey@0"
I20260925 10:32:27.408615 3674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0182845d45974fe8b8a5097258982ed3"
format_stamp: "Formatted at 2026-09-25 10:32:27 on dist-test-slave-dxdd"
server_key: "ca695bb899e2a6ceeb3577297211d5c6"
server_key_iv: "e3e2c2fe40af7c21944d43311b09398f"
server_key_version: "encryptionkey@0"
I20260925 10:32:27.413370 3674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 10:32:27.416412 3689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:27.417707 3674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:32:27.417850 3674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "0182845d45974fe8b8a5097258982ed3"
format_stamp: "Formatted at 2026-09-25 10:32:27 on dist-test-slave-dxdd"
server_key: "ca695bb899e2a6ceeb3577297211d5c6"
server_key_iv: "e3e2c2fe40af7c21944d43311b09398f"
server_key_version: "encryptionkey@0"
I20260925 10:32:27.418012 3674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:27.432277 3674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:27.435627 3674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:27.435887 3674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:27.436690 3674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:27.437873 3674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:27.437949 3674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:27.438133 3674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:27.438194 3674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:27.469393 3674 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42699
I20260925 10:32:27.469429 3801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42699 every 8 connection(s)
I20260925 10:32:27.470608 3674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:32:27.479564 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 3674
I20260925 10:32:27.479732 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:32:27.480237 388 external_mini_cluster.cc:1428] Setting key e0437192b3c88ce4c11f5d03583bffec
Sep 25 10:32:27 dist-test-slave-dxdd krb5kdc[3434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:27.492470 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:27.472895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60269 (local address 127.0.97.62:42649)
0925 10:32:27.473069 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:27.473073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:27.474086 (+ 1013us) server_negotiation.cc:407] Connection header received
0925 10:32:27.475856 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:27.475862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:27.475944 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:27.476048 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:27.477452 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:27.478604 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:27.480713 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:27.481350 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:27.481478 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:27.485812 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:27.485822 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:27.485830 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:27.485874 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:27.488972 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:27.489820 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:27.489828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:27.489835 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:27.490004 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:27.490482 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:27.490489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:27.490496 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:27.490877 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:27.491024 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:27.491793 (+ 769us) server_negotiation.cc:299] Negotiation successful
0925 10:32:27.492047 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:32:27.493314 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:27.473276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42649 (local address 127.0.97.2:60269)
0925 10:32:27.473933 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:27.473980 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:27.475599 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:27.476276 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:27.476288 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:27.476790 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:27.477284 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:27.477300 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:27.478745 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:27.478748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:27.480549 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:27.480564 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:27.481674 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:27.481684 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:27.481999 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:27.482764 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:27.482799 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:27.485583 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:27.489094 (+ 3511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:27.489101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:27.489115 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:27.489690 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:27.490125 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:27.490129 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:27.490133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:27.490372 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:27.491007 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:27.491014 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:27.491386 (+ 372us) client_negotiation.cc:769] Sending connection context
0925 10:32:27.491653 (+ 267us) client_negotiation.cc:240] Negotiation successful
0925 10:32:27.492115 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":261,"threads_started":1}
I20260925 10:32:27.494972 3802 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42649
I20260925 10:32:27.495281 3802 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:27.496124 3802 heartbeater.cc:511] Master 127.0.97.62:42649 requested a full tablet report, sending...
I20260925 10:32:27.497560 3479 ts_manager.cc:194] Registered new tserver with Master: 0182845d45974fe8b8a5097258982ed3 (127.0.97.2:42699)
I20260925 10:32:27.498411 3479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60269
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:32:27.523233 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42649
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:27.828594 3807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:27.828922 3807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:27.829131 3807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:27.836028 3807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:27.836141 3807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:27.836299 3807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:32:27.846899 3807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:27.848132 3807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:27.849515 3807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:27.862860 3812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:27.862833 3815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:27.862792 3813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:27.863592 3807 server_base.cc:1034] running on GCE node
I20260925 10:32:27.864176 3807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:27.865129 3807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:27.866353 3807 hybrid_clock.cc:648] HybridClock initialized: now 1790332347866271 us; error 43 us; skew 500 ppm
I20260925 10:32:27.866658 3807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:27 dist-test-slave-dxdd krb5kdc[3434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:27.869619 3807 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:32:27.870908 3807 webserver.cc:460] Webserver started at http://127.0.97.3:39393/ using document root <none> and password file <none>
I20260925 10:32:27.871469 3807 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:27.871572 3807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:27.871794 3807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:27.874413 3807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "7d8c60683b0a44df87017529ecbd7a6f"
format_stamp: "Formatted at 2026-09-25 10:32:27 on dist-test-slave-dxdd"
server_key: "b31fa5928921045b0e71e645af8f4192"
server_key_iv: "9ae85bd97775ccdc3cd4d603eee81a8d"
server_key_version: "encryptionkey@0"
I20260925 10:32:27.875100 3807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7d8c60683b0a44df87017529ecbd7a6f"
format_stamp: "Formatted at 2026-09-25 10:32:27 on dist-test-slave-dxdd"
server_key: "b31fa5928921045b0e71e645af8f4192"
server_key_iv: "9ae85bd97775ccdc3cd4d603eee81a8d"
server_key_version: "encryptionkey@0"
I20260925 10:32:27.879666 3807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 10:32:27.882659 3822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:27.883816 3807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 10:32:27.883976 3807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "7d8c60683b0a44df87017529ecbd7a6f"
format_stamp: "Formatted at 2026-09-25 10:32:27 on dist-test-slave-dxdd"
server_key: "b31fa5928921045b0e71e645af8f4192"
server_key_iv: "9ae85bd97775ccdc3cd4d603eee81a8d"
server_key_version: "encryptionkey@0"
I20260925 10:32:27.884155 3807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:27.899175 3807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:27.902555 3807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:27.902814 3807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:27.903676 3807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:27.904857 3807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:27.904955 3807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:27.905105 3807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:27.905176 3807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:27.937130 3807 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33347
I20260925 10:32:27.937175 3934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33347 every 8 connection(s)
I20260925 10:32:27.938355 3807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:32:27.947744 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 3807
I20260925 10:32:27.947902 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:32:27.948345 388 external_mini_cluster.cc:1428] Setting key 99358fb8a30b2e71245bcc6f85a56bb8
Sep 25 10:32:27 dist-test-slave-dxdd krb5kdc[3434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:27.956826 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:27.940942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57911 (local address 127.0.97.62:42649)
0925 10:32:27.941128 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:27.941133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:27.942188 (+ 1055us) server_negotiation.cc:407] Connection header received
0925 10:32:27.943900 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:27.943904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:27.943978 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:27.944064 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:27.945400 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:27.946094 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:27.947654 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:27.948190 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:27.948320 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:27.952067 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:27.952071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:27.952075 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:27.952105 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:27.954427 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:27.955041 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:27.955044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:27.955046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:27.955155 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:27.955507 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:27.955510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:27.955512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:27.955757 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:27.955856 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:27.956410 (+ 554us) server_negotiation.cc:299] Negotiation successful
0925 10:32:27.956597 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:32:27.957687 3937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:27.941353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42649 (local address 127.0.97.3:57911)
0925 10:32:27.942035 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:27.942078 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:27.943669 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:27.944263 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:27.944271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:27.944728 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:27.945257 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:27.945272 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:27.946227 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:27.946230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:27.947507 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:27.947518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:27.948416 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:27.948436 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:27.948674 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:27.949438 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:27.949466 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:27.951894 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:27.954560 (+ 2666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:27.954566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:27.954576 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:27.954939 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:27.955237 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:27.955240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:27.955242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:27.955395 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:27.955848 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:27.955853 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:27.956128 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 10:32:27.956322 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 10:32:27.956667 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":267,"threads_started":1}
I20260925 10:32:27.959049 3935 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42649
I20260925 10:32:27.959322 3935 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:27.959998 3935 heartbeater.cc:511] Master 127.0.97.62:42649 requested a full tablet report, sending...
I20260925 10:32:27.961293 3479 ts_manager.cc:194] Registered new tserver with Master: 7d8c60683b0a44df87017529ecbd7a6f (127.0.97.3:33347)
I20260925 10:32:27.962227 3479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57911
I20260925 10:32:27.963442 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:32:27 dist-test-slave-dxdd krb5kdc[3434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332347, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 10:32:27 dist-test-slave-dxdd krb5kdc[3434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332347, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:28.008859 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:27.995860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.0.97.62:42649)
0925 10:32:27.996025 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:27.996030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:27.996552 (+ 522us) server_negotiation.cc:407] Connection header received
0925 10:32:27.996706 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:27.996710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:27.996778 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:27.996861 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:27.998058 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:27.998823 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:27.999987 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:28.000368 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:28.000459 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:28.004022 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:28.004026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:28.004030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:28.004060 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:28.006360 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:28.006979 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:28.006985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:28.006989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:28.007116 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:28.007488 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:28.007495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:28.007498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:28.007790 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:28.008080 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:28.008346 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 10:32:28.008566 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 10:32:28.009342 3479 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45426
I20260925 10:32:28.010847 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 3540
Sep 25 10:32:28 dist-test-slave-dxdd krb5kdc[3434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332347, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:32:28.026623 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:28.011889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.0.97.1:35043)
0925 10:32:28.012418 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:28.012425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:28.012448 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:32:28.012496 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:28.012501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:28.012703 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:28.012819 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:28.013940 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:28.015600 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:28.016740 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:28.017575 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:28.017711 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:28.021385 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:28.021394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:28.021405 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:28.021432 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:28.024084 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:28.024704 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:28.024715 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:28.024722 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:28.024830 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:28.025182 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:28.025187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:28.025190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:28.025484 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:28.025771 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:28.026027 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 10:32:28.026281 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":274,"threads_started":1}
W20260925 10:32:28.027812 3663 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47016
W20260925 10:32:28.028913 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 3540: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:32:28.029212 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 3540
I20260925 10:32:28.044655 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 3674
Sep 25 10:32:28 dist-test-slave-dxdd krb5kdc[3434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332347, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:32:28.059332 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:28.045490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51482 (local address 127.0.97.2:42699)
0925 10:32:28.046081 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:28.046088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:28.046113 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:32:28.046169 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:28.046175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:28.046443 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:28.046575 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:28.047691 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:28.048688 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:28.049836 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:28.050423 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:28.050514 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:28.054192 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:28.054201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:28.054213 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:28.054241 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:28.057151 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:28.057683 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:28.057689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:28.057691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:28.057794 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:28.058100 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:28.058103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:28.058105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:28.058448 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:28.058642 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:28.058885 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 10:32:28.059067 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":318,"threads_started":1}
W20260925 10:32:28.060456 3797 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51482
W20260925 10:32:28.061429 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 3674: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:32:28.061579 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 3674
I20260925 10:32:28.076267 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 3807
Sep 25 10:32:28 dist-test-slave-dxdd krb5kdc[3434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332347, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:32:28.091741 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:28.077137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.0.97.3:33347)
0925 10:32:28.077766 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:28.077773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:28.077798 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:32:28.077850 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:28.077856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:28.078053 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:28.078220 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:28.079469 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:28.080467 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:28.081647 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:28.082173 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:28.082308 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:28.086225 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:28.086235 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:28.086247 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:28.086277 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:28.088992 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:28.089619 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:28.089626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:28.089628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:28.089761 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:28.090126 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:28.090132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:28.090136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:28.090527 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:28.090835 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:28.091150 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 10:32:28.091403 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":327,"threads_started":1}
W20260925 10:32:28.092878 3930 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45414
W20260925 10:32:28.093926 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 3807: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:32:28.094071 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 3807
I20260925 10:32:28.109274 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 3447
I20260925 10:32:28.110272 3509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:28.209354 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 3447
2026-09-25T10:32:28Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3380 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:32:28 dist-test-slave-dxdd krb5kdc[3952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:32:28 dist-test-slave-dxdd krb5kdc[3952](info): set up 2 sockets
Sep 25 10:32:28 dist-test-slave-dxdd krb5kdc[3952](info): commencing operation
krb5kdc: starting...
W20260925 10:32:29.294680 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:32:29 dist-test-slave-dxdd krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:32:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:32:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:32:29.480003 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42201
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:29.787638 3965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:29.787895 3965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:29.788066 3965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:29.794963 3965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:29.795060 3965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:29.795163 3965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:29.795257 3965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:29.795336 3965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:32:29.804359 3965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:29.805536 3965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:29.806824 3965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:29.818171 3970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:29.818246 3971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:29.818219 3973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:29.819242 3965 server_base.cc:1034] running on GCE node
I20260925 10:32:29.819945 3965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:29.821187 3965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:29.822399 3965 hybrid_clock.cc:648] HybridClock initialized: now 1790332349822362 us; error 46 us; skew 500 ppm
I20260925 10:32:29.822721 3965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:29 dist-test-slave-dxdd krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:29.832320 3965 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:32:29.833734 3965 webserver.cc:460] Webserver started at http://127.0.97.62:34283/ using document root <none> and password file <none>
I20260925 10:32:29.834427 3965 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:29.834560 3965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:29.834854 3965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:29.837626 3965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "230eb100e98f4e2db3188aac5a847d50"
format_stamp: "Formatted at 2026-09-25 10:32:29 on dist-test-slave-dxdd"
server_key: "507087b4d859e2c6f72f6dc75d7e18db"
server_key_iv: "233e46a36a032a809e5d7a04c0a3bce7"
server_key_version: "encryptionkey@0"
I20260925 10:32:29.838421 3965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "230eb100e98f4e2db3188aac5a847d50"
format_stamp: "Formatted at 2026-09-25 10:32:29 on dist-test-slave-dxdd"
server_key: "507087b4d859e2c6f72f6dc75d7e18db"
server_key_iv: "233e46a36a032a809e5d7a04c0a3bce7"
server_key_version: "encryptionkey@0"
I20260925 10:32:29.843133 3965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 10:32:29.846239 3980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:29.847554 3965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:32:29.847729 3965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "230eb100e98f4e2db3188aac5a847d50"
format_stamp: "Formatted at 2026-09-25 10:32:29 on dist-test-slave-dxdd"
server_key: "507087b4d859e2c6f72f6dc75d7e18db"
server_key_iv: "233e46a36a032a809e5d7a04c0a3bce7"
server_key_version: "encryptionkey@0"
I20260925 10:32:29.847929 3965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:29.862987 3965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:29.866604 3965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:29.866858 3965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:29.884419 3965 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42201
I20260925 10:32:29.884454 4031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42201 every 8 connection(s)
I20260925 10:32:29.885811 3965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:32:29.889328 4032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:29.895375 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 3965
I20260925 10:32:29.895599 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:32:29.896328 388 external_mini_cluster.cc:1428] Setting key 7a5aad9ef273c8ecdd0547ed775432f1
I20260925 10:32:29.899649 4032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50: Bootstrap starting.
I20260925 10:32:29.902895 4032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:29.903898 4032 log.cc:824] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:29.906824 4032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50: No bootstrap required, opened a new log
Sep 25 10:32:29 dist-test-slave-dxdd krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:29.917028 4032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230eb100e98f4e2db3188aac5a847d50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42201 } }
I20260925 10:32:29.917438 4032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:29.917555 4032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230eb100e98f4e2db3188aac5a847d50, State: Initialized, Role: FOLLOWER
I20260925 10:32:29.918195 4032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230eb100e98f4e2db3188aac5a847d50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42201 } }
I20260925 10:32:29.918515 4032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:32:29.918649 4032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:32:29.918829 4032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:29.919819 4035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:29.898269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.0.97.62:42201)
0925 10:32:29.898961 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:29.898971 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:29.899009 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:32:29.900658 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:29.900678 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:29.901031 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:29.901327 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:29.903125 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:29.904570 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:29.906252 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:29.907027 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:29.907150 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:29.911518 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:29.911529 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:29.911553 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:29.911602 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:29.915492 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:29.916145 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:29.916153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:29.916161 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:29.916346 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:29.916663 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:29.916667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:29.916670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:29.917311 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:29.917605 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:29.918025 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 10:32:29.918476 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":326,"threads_started":1}
I20260925 10:32:29.921399 4032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230eb100e98f4e2db3188aac5a847d50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42201 } }
I20260925 10:32:29.921880 4032 leader_election.cc:302] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 230eb100e98f4e2db3188aac5a847d50; no voters:
I20260925 10:32:29.922494 4032 leader_election.cc:288] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:32:29.922583 4037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:29.923321 4037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [term 1 LEADER]: Becoming Leader. State: Replica: 230eb100e98f4e2db3188aac5a847d50, State: Running, Role: LEADER
I20260925 10:32:29.923815 4037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230eb100e98f4e2db3188aac5a847d50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42201 } }
I20260925 10:32:29.924732 4032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:29.926503 4039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 230eb100e98f4e2db3188aac5a847d50. Latest consensus state: current_term: 1 leader_uuid: "230eb100e98f4e2db3188aac5a847d50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230eb100e98f4e2db3188aac5a847d50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42201 } } }
I20260925 10:32:29.926849 4039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:29.926615 4038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "230eb100e98f4e2db3188aac5a847d50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230eb100e98f4e2db3188aac5a847d50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42201 } } }
I20260925 10:32:29.927134 4038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:29.929749 4043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:29.934530 4043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:29.943004 4043 catalog_manager.cc:1347] Generated new cluster ID: 60b0375819c64b5d809152693ebb17f1
I20260925 10:32:29.943115 4043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:29.965257 4043 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:32:29.966651 4043 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:29.977278 4043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 230eb100e98f4e2db3188aac5a847d50: Generated new TSK 0
I20260925 10:32:29.978206 4043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:32:30.025619 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42201
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:30.335462 4058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:30.335721 4058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:30.335846 4058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:30.342800 4058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:30.342902 4058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:30.343014 4058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:32:30.353047 4058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:30.354178 4058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:30.355417 4058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:30.368172 4063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:30.368206 4064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:30.368290 4066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:30.368957 4058 server_base.cc:1034] running on GCE node
I20260925 10:32:30.369485 4058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:30.370282 4058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:30.371528 4058 hybrid_clock.cc:648] HybridClock initialized: now 1790332350371478 us; error 47 us; skew 500 ppm
I20260925 10:32:30.371773 4058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:30 dist-test-slave-dxdd krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:30.374624 4058 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:32:30.375838 4058 webserver.cc:460] Webserver started at http://127.0.97.1:37957/ using document root <none> and password file <none>
I20260925 10:32:30.376434 4058 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:30.376528 4058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:30.376757 4058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:30.379467 4058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "a396a724d1514724b77a0d46131b4a5e"
format_stamp: "Formatted at 2026-09-25 10:32:30 on dist-test-slave-dxdd"
server_key: "02d3499e6906b509a9ee8acfb4cad0e9"
server_key_iv: "402803710033910cac8a9c1c2054f4af"
server_key_version: "encryptionkey@0"
I20260925 10:32:30.380147 4058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a396a724d1514724b77a0d46131b4a5e"
format_stamp: "Formatted at 2026-09-25 10:32:30 on dist-test-slave-dxdd"
server_key: "02d3499e6906b509a9ee8acfb4cad0e9"
server_key_iv: "402803710033910cac8a9c1c2054f4af"
server_key_version: "encryptionkey@0"
I20260925 10:32:30.384701 4058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 10:32:30.387807 4073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:30.388993 4058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:32:30.389154 4058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "a396a724d1514724b77a0d46131b4a5e"
format_stamp: "Formatted at 2026-09-25 10:32:30 on dist-test-slave-dxdd"
server_key: "02d3499e6906b509a9ee8acfb4cad0e9"
server_key_iv: "402803710033910cac8a9c1c2054f4af"
server_key_version: "encryptionkey@0"
I20260925 10:32:30.389338 4058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:30.406072 4058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:30.409618 4058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:30.409883 4058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:30.410799 4058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:30.412019 4058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:30.412127 4058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:30.412267 4058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:30.412336 4058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:30.444713 4058 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43827
I20260925 10:32:30.444763 4185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43827 every 8 connection(s)
I20260925 10:32:30.445945 4058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:32:30.451947 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 4058
I20260925 10:32:30.452181 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:32:30.452988 388 external_mini_cluster.cc:1428] Setting key 28f963b4432c9f2383c4a0e59ee0fac3
Sep 25 10:32:30 dist-test-slave-dxdd krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:30.466297 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:30.448286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34295 (local address 127.0.97.62:42201)
0925 10:32:30.448828 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:30.448836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:30.449894 (+ 1058us) server_negotiation.cc:407] Connection header received
0925 10:32:30.451756 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:30.451760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:30.451822 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:30.451914 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:30.453864 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:30.454769 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:30.456299 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:30.457088 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:30.457224 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:30.460710 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:30.460714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:30.460718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:30.460747 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:30.463294 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:30.464180 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:30.464184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:30.464186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:30.464312 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:30.464768 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:30.464771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:30.464773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:30.465020 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:30.465123 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:30.465918 (+ 795us) server_negotiation.cc:299] Negotiation successful
0925 10:32:30.466062 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":248,"threads_started":1}
I20260925 10:32:30.467891 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:30.448866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42201 (local address 127.0.97.1:34295)
0925 10:32:30.449736 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:30.449784 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:30.451430 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:30.452205 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:30.452217 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:30.452959 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:30.453696 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:30.453714 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:30.454943 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:30.454947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:30.456146 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:30.456157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:30.457288 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:30.457292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:30.457466 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:30.458092 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:30.458115 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:30.460510 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:30.463444 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:30.463455 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:30.463472 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:30.464048 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:30.464434 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:30.464441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:30.464448 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:30.464667 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:30.465142 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:30.465152 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:30.465565 (+ 413us) client_negotiation.cc:769] Sending connection context
0925 10:32:30.465864 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 10:32:30.466449 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":357,"threads_started":1}
I20260925 10:32:30.469451 4186 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42201
I20260925 10:32:30.469727 4186 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:30.470569 4186 heartbeater.cc:511] Master 127.0.97.62:42201 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260925 10:32:30.472718 3997 ts_manager.cc:194] Registered new tserver with Master: a396a724d1514724b77a0d46131b4a5e (127.0.97.1:43827)
I20260925 10:32:30.474200 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:32:30.496678 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42201
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:30.808614 4192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:30.808924 4192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:30.809077 4192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:30.816468 4192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:30.816574 4192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:30.816788 4192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:32:30.826788 4192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:30.828066 4192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:30.829360 4192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:30.842635 4197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:30.842705 4198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:30.842681 4200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:30.843490 4192 server_base.cc:1034] running on GCE node
I20260925 10:32:30.844173 4192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:30.845068 4192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:30.846455 4192 hybrid_clock.cc:648] HybridClock initialized: now 1790332350846372 us; error 68 us; skew 500 ppm
I20260925 10:32:30.846776 4192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:30 dist-test-slave-dxdd krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:30.850423 4192 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:32:30.851969 4192 webserver.cc:460] Webserver started at http://127.0.97.2:35839/ using document root <none> and password file <none>
I20260925 10:32:30.852695 4192 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:30.852830 4192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:30.853127 4192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:30.855873 4192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "4802590dd0b04c05ac2d863dcebb4964"
format_stamp: "Formatted at 2026-09-25 10:32:30 on dist-test-slave-dxdd"
server_key: "f152233baf1143d500d81021954e6707"
server_key_iv: "2d1c8c6176b2e3677fa9926621b6f1ad"
server_key_version: "encryptionkey@0"
I20260925 10:32:30.856606 4192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4802590dd0b04c05ac2d863dcebb4964"
format_stamp: "Formatted at 2026-09-25 10:32:30 on dist-test-slave-dxdd"
server_key: "f152233baf1143d500d81021954e6707"
server_key_iv: "2d1c8c6176b2e3677fa9926621b6f1ad"
server_key_version: "encryptionkey@0"
I20260925 10:32:30.861353 4192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 10:32:30.864483 4207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:30.865715 4192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:32:30.865875 4192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "4802590dd0b04c05ac2d863dcebb4964"
format_stamp: "Formatted at 2026-09-25 10:32:30 on dist-test-slave-dxdd"
server_key: "f152233baf1143d500d81021954e6707"
server_key_iv: "2d1c8c6176b2e3677fa9926621b6f1ad"
server_key_version: "encryptionkey@0"
I20260925 10:32:30.866062 4192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:30.889707 4192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:30.894155 4192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:30.894493 4192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:30.895438 4192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:30.896672 4192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:30.896772 4192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:30.896932 4192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:30.897001 4192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:30.929345 4192 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37173
I20260925 10:32:30.929399 4319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37173 every 8 connection(s)
I20260925 10:32:30.930706 4192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:32:30.932358 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 4192
I20260925 10:32:30.932555 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:32:30.933010 388 external_mini_cluster.cc:1428] Setting key db780911853b69ff2af23a0bbf644d2d
Sep 25 10:32:30 dist-test-slave-dxdd krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:30.952732 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:30.933249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44827 (local address 127.0.97.62:42201)
0925 10:32:30.933468 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:30.933476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:30.934670 (+ 1194us) server_negotiation.cc:407] Connection header received
0925 10:32:30.937322 (+ 2652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:30.937326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:30.937378 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:30.937458 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:30.939515 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:30.940227 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:30.942061 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:30.942496 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:30.942590 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:30.945990 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:30.945999 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:30.946007 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:30.946049 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:30.949578 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:30.950247 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:30.950253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:30.950258 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:30.950447 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:30.950871 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:30.950879 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:30.950885 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:30.951257 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:30.951441 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:30.952071 (+ 630us) server_negotiation.cc:299] Negotiation successful
0925 10:32:30.952325 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:32:30.953533 4322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:30.933710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42201 (local address 127.0.97.2:44827)
0925 10:32:30.934524 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:30.934562 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:30.937046 (+ 2484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:30.937727 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:30.937740 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:30.938548 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:30.939347 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:30.939373 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:30.940377 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:30.940385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:30.941915 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:30.941926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:30.942672 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:30.942677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:30.942859 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:30.943553 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:30.943574 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:30.945793 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:30.949709 (+ 3916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:30.949716 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:30.949727 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:30.950124 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:30.950551 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:30.950554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:30.950557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:30.950714 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:30.951393 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:30.951423 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:30.951725 (+ 302us) client_negotiation.cc:769] Sending connection context
0925 10:32:30.951947 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 10:32:30.952398 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":345,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260925 10:32:30.955080 4320 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42201
I20260925 10:32:30.955428 4320 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260925 10:32:30.956224 4320 heartbeater.cc:511] Master 127.0.97.62:42201 requested a full tablet report, sending...
I20260925 10:32:30.957652 3997 ts_manager.cc:194] Registered new tserver with Master: 4802590dd0b04c05ac2d863dcebb4964 (127.0.97.2:37173)
I20260925 10:32:30.958503 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:32:30.977648 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42201
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:31.289389 4325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:31.289620 4325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:31.289707 4325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:31.296586 4325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:31.296674 4325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:31.296775 4325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:32:31.305773 4325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:31.306836 4325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:31.308007 4325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:31.320362 4333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:31.320441 4331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:31.320885 4330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:31.321543 4325 server_base.cc:1034] running on GCE node
I20260925 10:32:31.322023 4325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:31.322958 4325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:31.324164 4325 hybrid_clock.cc:648] HybridClock initialized: now 1790332351324114 us; error 45 us; skew 500 ppm
I20260925 10:32:31.324422 4325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:31 dist-test-slave-dxdd krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:31.327432 4325 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:32:31.328701 4325 webserver.cc:460] Webserver started at http://127.0.97.3:35547/ using document root <none> and password file <none>
I20260925 10:32:31.329344 4325 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:31.329488 4325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:31.329777 4325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:31.332646 4325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "cec2990233fc48688d619f06fcd9b209"
format_stamp: "Formatted at 2026-09-25 10:32:31 on dist-test-slave-dxdd"
server_key: "9b4b990629e50687388f9dcce96bfefb"
server_key_iv: "be18f9362b58b8444edde36d364f5413"
server_key_version: "encryptionkey@0"
I20260925 10:32:31.333379 4325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cec2990233fc48688d619f06fcd9b209"
format_stamp: "Formatted at 2026-09-25 10:32:31 on dist-test-slave-dxdd"
server_key: "9b4b990629e50687388f9dcce96bfefb"
server_key_iv: "be18f9362b58b8444edde36d364f5413"
server_key_version: "encryptionkey@0"
I20260925 10:32:31.338002 4325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260925 10:32:31.341090 4340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:31.342242 4325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:32:31.342437 4325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "cec2990233fc48688d619f06fcd9b209"
format_stamp: "Formatted at 2026-09-25 10:32:31 on dist-test-slave-dxdd"
server_key: "9b4b990629e50687388f9dcce96bfefb"
server_key_iv: "be18f9362b58b8444edde36d364f5413"
server_key_version: "encryptionkey@0"
I20260925 10:32:31.342628 4325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:31.366436 4325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:31.370069 4325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:31.370393 4325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:31.371306 4325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:31.372516 4325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:31.372586 4325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:31.372663 4325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:31.372705 4325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:31.404532 4325 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43077
I20260925 10:32:31.404616 4452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43077 every 8 connection(s)
I20260925 10:32:31.405742 4325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:32:31.414145 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 4325
I20260925 10:32:31.414345 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:32:31.414770 388 external_mini_cluster.cc:1428] Setting key b161b32c03cf2cad12a5b7e6c341d4d1
Sep 25 10:32:31 dist-test-slave-dxdd krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:31.424396 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:31.408138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43087 (local address 127.0.97.62:42201)
0925 10:32:31.408347 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:31.408352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:31.409460 (+ 1108us) server_negotiation.cc:407] Connection header received
0925 10:32:31.411216 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:31.411220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:31.411280 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:31.411364 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:31.412790 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:31.413481 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:31.415166 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:31.415761 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:31.415913 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:31.419442 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:31.419447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:31.419450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:31.419477 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:31.421978 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:31.422665 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:31.422669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:31.422671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:31.422779 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:31.423125 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:31.423128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:31.423130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:31.423368 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:31.423480 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:31.424025 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 10:32:31.424183 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 10:32:31.425314 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:31.408589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42201 (local address 127.0.97.3:43087)
0925 10:32:31.409302 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:31.409341 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:31.410982 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:31.411554 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:31.411562 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:31.412110 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:31.412645 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:31.412661 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:31.413603 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:31.413606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:31.415024 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:31.415035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:31.416290 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:31.416297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:31.416584 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:31.417227 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:31.417255 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:31.419268 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:31.422091 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:31.422096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:31.422105 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:31.422567 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:31.422868 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:31.422871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:31.422875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:31.423023 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:31.423476 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:31.423481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:31.423734 (+ 253us) client_negotiation.cc:769] Sending connection context
0925 10:32:31.423930 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 10:32:31.424260 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":305,"threads_started":1}
I20260925 10:32:31.426533 4453 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42201
I20260925 10:32:31.426772 4453 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:31.427346 4453 heartbeater.cc:511] Master 127.0.97.62:42201 requested a full tablet report, sending...
I20260925 10:32:31.428718 3997 ts_manager.cc:194] Registered new tserver with Master: cec2990233fc48688d619f06fcd9b209 (127.0.97.3:43077)
I20260925 10:32:31.429725 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:32:31.429744 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43087
Sep 25 10:32:31 dist-test-slave-dxdd krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:32:31 dist-test-slave-dxdd krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:32:31.475649 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:31.462175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.0.97.1:43827)
0925 10:32:31.462766 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:31.462773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:31.462799 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:32:31.462858 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:31.462864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:31.463041 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:31.463164 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:31.464434 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:31.465373 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:31.466516 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:31.467008 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:31.467100 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:31.470753 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:31.470762 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:31.470774 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:31.470801 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:31.473399 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:31.473928 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:31.473934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:31.473937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:31.474051 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:31.474380 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:31.474384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:31.474386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:31.474681 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:31.474882 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:31.475084 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 10:32:31.475250 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":342,"threads_started":1}
I20260925 10:32:31.477113 4186 heartbeater.cc:503] Master 127.0.97.62:42201 was elected leader, sending a full tablet report...
Sep 25 10:32:31 dist-test-slave-dxdd krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:31.491539 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:31.479811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.0.97.62:42201)
0925 10:32:31.480063 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:31.480068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:31.480205 (+ 137us) server_negotiation.cc:407] Connection header received
0925 10:32:31.480313 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:31.480316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:31.480367 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:31.480443 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:31.481510 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:31.482172 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:31.483344 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:31.483725 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:31.483814 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:31.487204 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:31.487208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:31.487211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:31.487238 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:31.489404 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:31.489933 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:31.489936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:31.489939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:31.490053 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:31.490407 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:31.490410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:31.490412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:31.490663 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:31.490866 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:31.491107 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 10:32:31.491238 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260925 10:32:31.492048 3997 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43156
I20260925 10:32:31.492988 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 4058
I20260925 10:32:31.503782 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:31.493737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.0.97.1:43827)
0925 10:32:31.493900 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:31.493905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:31.494180 (+ 275us) server_negotiation.cc:407] Connection header received
0925 10:32:31.494288 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:31.494291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:31.494395 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:31.494470 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:31.495478 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:31.496115 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:31.497171 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:31.497560 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:31.497651 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:31.499656 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:31.499661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:31.499664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:31.499689 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:31.501898 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:31.502489 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:31.502492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:31.502494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:31.502609 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:31.502885 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:31.502887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:31.502889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:31.503088 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:31.503249 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:31.503430 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 10:32:31.503563 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:32:31.504335 4181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:31.586773 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 4058
I20260925 10:32:31.604667 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 4192
Sep 25 10:32:31 dist-test-slave-dxdd krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:32:31.619167 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:31.605436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.0.97.2:37173)
0925 10:32:31.605985 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:31.605998 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:31.606025 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:32:31.606083 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:31.606089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:31.606297 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:31.606490 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:31.607511 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:31.608473 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:31.609624 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:31.610134 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:31.610223 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:31.613999 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:31.614009 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:31.614024 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:31.614052 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:31.616886 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:31.617444 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:31.617451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:31.617454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:31.617563 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:31.617853 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:31.617855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:31.617857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:31.618153 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:31.618601 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:31.618716 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 10:32:31.618896 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":290,"threads_started":1}
I20260925 10:32:31.620348 4315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:31.700959 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 4192
I20260925 10:32:31.716321 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 4325
Sep 25 10:32:31 dist-test-slave-dxdd krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:32:31.731511 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:31.717170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.0.97.3:43077)
0925 10:32:31.717705 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:31.717711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:31.717738 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:32:31.717793 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:31.717799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:31.718000 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:31.718166 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:31.719412 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:31.720367 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:31.721602 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:31.722123 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:31.722253 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:31.726381 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:31.726391 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:31.726404 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:31.726435 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:31.729098 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:31.729616 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:31.729622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:31.729624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:31.729729 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:31.730023 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:31.730034 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:31.730037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:31.730447 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:31.730691 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:31.730895 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 10:32:31.731099 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":256,"threads_started":1}
I20260925 10:32:31.732666 4448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:31.812110 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 4325
I20260925 10:32:31.827744 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 3965
I20260925 10:32:31.828893 4027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:31.927580 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 3965
2026-09-25T10:32:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3714 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:32:32 dist-test-slave-dxdd krb5kdc[4472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:32:32 dist-test-slave-dxdd krb5kdc[4472](info): set up 2 sockets
Sep 25 10:32:32 dist-test-slave-dxdd krb5kdc[4472](info): commencing operation
krb5kdc: starting...
W20260925 10:32:33.006712 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:32:33 dist-test-slave-dxdd krb5kdc[4472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:32:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:32:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:32:33.190999 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42345
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:33.493954 4485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:33.494267 4485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:33.494459 4485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:33.501247 4485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:33.501394 4485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:33.501498 4485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:33.501607 4485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:33.501679 4485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:32:33.510751 4485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:33.511919 4485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:33.513131 4485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:33.523970 4490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:33.523938 4491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:33.523990 4493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:33.524967 4485 server_base.cc:1034] running on GCE node
I20260925 10:32:33.525707 4485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:33.526947 4485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:33.528131 4485 hybrid_clock.cc:648] HybridClock initialized: now 1790332353528070 us; error 54 us; skew 500 ppm
I20260925 10:32:33.528456 4485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:33 dist-test-slave-dxdd krb5kdc[4472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:33.531493 4485 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:32:33.532568 4485 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:33.532672 4485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:33.532979 4485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:33.535775 4485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "9fcdf46fd3d4462a902f2294cf306bc1"
format_stamp: "Formatted at 2026-09-25 10:32:33 on dist-test-slave-dxdd"
server_key: "78fc65044959044409241cf428c71cd8"
server_key_iv: "3607c3d8b517fa89ba1511cf380c07ca"
server_key_version: "encryptionkey@0"
I20260925 10:32:33.536517 4485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "9fcdf46fd3d4462a902f2294cf306bc1"
format_stamp: "Formatted at 2026-09-25 10:32:33 on dist-test-slave-dxdd"
server_key: "78fc65044959044409241cf428c71cd8"
server_key_iv: "3607c3d8b517fa89ba1511cf380c07ca"
server_key_version: "encryptionkey@0"
I20260925 10:32:33.541313 4485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 10:32:33.544467 4499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:33.545656 4485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:32:33.545815 4485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "9fcdf46fd3d4462a902f2294cf306bc1"
format_stamp: "Formatted at 2026-09-25 10:32:33 on dist-test-slave-dxdd"
server_key: "78fc65044959044409241cf428c71cd8"
server_key_iv: "3607c3d8b517fa89ba1511cf380c07ca"
server_key_version: "encryptionkey@0"
I20260925 10:32:33.546015 4485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:33.566109 4485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:33.570070 4485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:33.570348 4485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:33.587535 4485 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42345
I20260925 10:32:33.587556 4550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42345 every 8 connection(s)
I20260925 10:32:33.588367 4485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:32:33.591677 4551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:33.595821 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 4485
I20260925 10:32:33.596047 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:32:33.596802 388 external_mini_cluster.cc:1428] Setting key 52d64f2e63732e6e230e36de02ed36f2
I20260925 10:32:33.602672 4551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1: Bootstrap starting.
I20260925 10:32:33.606030 4551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:33.607141 4551 log.cc:824] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:33.610216 4551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1: No bootstrap required, opened a new log
Sep 25 10:32:33 dist-test-slave-dxdd krb5kdc[4472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:33.620226 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:33.598877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.0.97.62:42345)
0925 10:32:33.599698 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:33.599709 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:33.599757 (+ 48us) server_negotiation.cc:407] Connection header received
0925 10:32:33.601399 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:33.601418 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:33.601761 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:33.602059 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:33.603802 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:33.605067 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:33.606810 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:33.607489 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:33.607695 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:33.612038 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:33.612049 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:33.612065 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:33.612111 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:33.615951 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:33.616572 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:33.616586 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:33.616594 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:33.616793 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:33.617147 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:33.617154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:33.617158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:33.617880 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:33.618158 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:33.618501 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 10:32:33.618939 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":389,"threads_started":1}
I20260925 10:32:33.620740 4551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcdf46fd3d4462a902f2294cf306bc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42345 } }
I20260925 10:32:33.621102 4551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:33.621214 4551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fcdf46fd3d4462a902f2294cf306bc1, State: Initialized, Role: FOLLOWER
I20260925 10:32:33.621815 4551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcdf46fd3d4462a902f2294cf306bc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42345 } }
I20260925 10:32:33.622056 4551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:32:33.622200 4551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:32:33.622431 4551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:33.624570 4551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcdf46fd3d4462a902f2294cf306bc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42345 } }
I20260925 10:32:33.625027 4551 leader_election.cc:302] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fcdf46fd3d4462a902f2294cf306bc1; no voters:
I20260925 10:32:33.625623 4551 leader_election.cc:288] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:32:33.625754 4556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:33.626873 4556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [term 1 LEADER]: Becoming Leader. State: Replica: 9fcdf46fd3d4462a902f2294cf306bc1, State: Running, Role: LEADER
I20260925 10:32:33.627372 4556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcdf46fd3d4462a902f2294cf306bc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42345 } }
I20260925 10:32:33.627779 4551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:33.629837 4558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fcdf46fd3d4462a902f2294cf306bc1. Latest consensus state: current_term: 1 leader_uuid: "9fcdf46fd3d4462a902f2294cf306bc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcdf46fd3d4462a902f2294cf306bc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42345 } } }
I20260925 10:32:33.630154 4558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:33.630748 4557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fcdf46fd3d4462a902f2294cf306bc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcdf46fd3d4462a902f2294cf306bc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42345 } } }
I20260925 10:32:33.631108 4557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:33.639616 4564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:33.643646 4564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:33.650619 4564 catalog_manager.cc:1347] Generated new cluster ID: 96ecc2db235040bdbd123055262c50d3
I20260925 10:32:33.650717 4564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:33.668321 4564 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:32:33.670107 4564 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:33.680527 4564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1: Generated new TSK 0
I20260925 10:32:33.681504 4564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:32:33.731570 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42345
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:34.046587 4578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:34.046832 4578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:34.046993 4578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:34.054190 4578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:34.054284 4578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:34.054504 4578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:32:34.063736 4578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:34.064970 4578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:34.066169 4578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:34.078339 4583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:34.078287 4584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:34.078433 4586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:34.079236 4578 server_base.cc:1034] running on GCE node
I20260925 10:32:34.079785 4578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:34.080677 4578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:34.081898 4578 hybrid_clock.cc:648] HybridClock initialized: now 1790332354081851 us; error 42 us; skew 500 ppm
I20260925 10:32:34.082190 4578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:34 dist-test-slave-dxdd krb5kdc[4472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:34.085223 4578 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:32:34.086535 4578 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:34.086663 4578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:34.086970 4578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:34.089699 4578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "8b59ac1d2ca74bd8819f00975c903b93"
format_stamp: "Formatted at 2026-09-25 10:32:34 on dist-test-slave-dxdd"
server_key: "747275e2834a41b336fdbe46293c650f"
server_key_iv: "91426675724b318383f10032cb4ea2f1"
server_key_version: "encryptionkey@0"
I20260925 10:32:34.090590 4578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8b59ac1d2ca74bd8819f00975c903b93"
format_stamp: "Formatted at 2026-09-25 10:32:34 on dist-test-slave-dxdd"
server_key: "747275e2834a41b336fdbe46293c650f"
server_key_iv: "91426675724b318383f10032cb4ea2f1"
server_key_version: "encryptionkey@0"
I20260925 10:32:34.095225 4578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:32:34.098516 4592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:34.099718 4578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:32:34.099893 4578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "8b59ac1d2ca74bd8819f00975c903b93"
format_stamp: "Formatted at 2026-09-25 10:32:34 on dist-test-slave-dxdd"
server_key: "747275e2834a41b336fdbe46293c650f"
server_key_iv: "91426675724b318383f10032cb4ea2f1"
server_key_version: "encryptionkey@0"
I20260925 10:32:34.100100 4578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:34.129062 4578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:34.133029 4578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:34.133308 4578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:34.134243 4578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:34.135529 4578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:34.135648 4578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:34.135775 4578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:34.135857 4578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:34.168633 4578 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43873
I20260925 10:32:34.168680 4704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43873 every 8 connection(s)
I20260925 10:32:34.169502 4578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:32:34.179888 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 4578
I20260925 10:32:34.180100 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:32:34.180935 388 external_mini_cluster.cc:1428] Setting key 5e585fc8a9606b991cd7946c03164f25
Sep 25 10:32:34 dist-test-slave-dxdd krb5kdc[4472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:34.190240 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:34.171907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33411 (local address 127.0.97.62:42345)
0925 10:32:34.172309 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:34.172315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:34.173581 (+ 1266us) server_negotiation.cc:407] Connection header received
0925 10:32:34.175357 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:34.175361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:34.175424 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:34.175524 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:34.176915 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:34.177629 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:34.179192 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:34.179578 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:34.179668 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:34.183618 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:34.183625 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:34.183631 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:34.183675 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:34.187229 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:34.187908 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:34.187916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:34.187929 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:34.188072 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:34.188443 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:34.188451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:34.188457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:34.188848 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:34.189035 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:34.189843 (+ 808us) server_negotiation.cc:299] Negotiation successful
0925 10:32:34.190008 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":187,"threads_started":1}
I20260925 10:32:34.191754 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:34.172518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42345 (local address 127.0.97.1:33411)
0925 10:32:34.173207 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:34.173250 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:34.175119 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:34.175718 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:34.175727 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:34.176213 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:34.176766 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:34.176782 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:34.177778 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:34.177782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:34.179063 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:34.179073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:34.179935 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:34.179940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:34.180212 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:34.180969 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:34.180995 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:34.183398 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:34.187362 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:34.187369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:34.187381 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:34.187776 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:34.188172 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:34.188175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:34.188178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:34.188330 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:34.189009 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:34.189018 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:34.189474 (+ 456us) client_negotiation.cc:769] Sending connection context
0925 10:32:34.189787 (+ 313us) client_negotiation.cc:240] Negotiation successful
0925 10:32:34.190378 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":313,"threads_started":1}
I20260925 10:32:34.193238 4705 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42345
I20260925 10:32:34.193531 4705 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:34.194166 4705 heartbeater.cc:511] Master 127.0.97.62:42345 requested a full tablet report, sending...
I20260925 10:32:34.196324 4516 ts_manager.cc:194] Registered new tserver with Master: 8b59ac1d2ca74bd8819f00975c903b93 (127.0.97.1:43873)
I20260925 10:32:34.198520 4516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33411
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:32:34.223443 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42345
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:34.523200 4711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:34.523440 4711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:34.523587 4711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:34.530623 4711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:34.530723 4711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:34.530922 4711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:32:34.540351 4711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:34.541544 4711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:34.542857 4711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:34.555002 4716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:34.554966 4717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:34.555040 4719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:34.555807 4711 server_base.cc:1034] running on GCE node
I20260925 10:32:34.556401 4711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:34.557263 4711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:34.558403 4711 hybrid_clock.cc:648] HybridClock initialized: now 1790332354558335 us; error 55 us; skew 500 ppm
I20260925 10:32:34.558709 4711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:34 dist-test-slave-dxdd krb5kdc[4472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:34.561846 4711 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:32:34.562960 4711 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:34.563072 4711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:34.563380 4711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:34.566043 4711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "b0d45977c89745729774336f604f617b"
format_stamp: "Formatted at 2026-09-25 10:32:34 on dist-test-slave-dxdd"
server_key: "67d9fc02e5f827a3be29590ed6b42f27"
server_key_iv: "e37cab7ba69825bbc9275b2eb0eed47a"
server_key_version: "encryptionkey@0"
I20260925 10:32:34.566875 4711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b0d45977c89745729774336f604f617b"
format_stamp: "Formatted at 2026-09-25 10:32:34 on dist-test-slave-dxdd"
server_key: "67d9fc02e5f827a3be29590ed6b42f27"
server_key_iv: "e37cab7ba69825bbc9275b2eb0eed47a"
server_key_version: "encryptionkey@0"
I20260925 10:32:34.571658 4711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:32:34.575596 4725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:34.576822 4711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:32:34.576959 4711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "b0d45977c89745729774336f604f617b"
format_stamp: "Formatted at 2026-09-25 10:32:34 on dist-test-slave-dxdd"
server_key: "67d9fc02e5f827a3be29590ed6b42f27"
server_key_iv: "e37cab7ba69825bbc9275b2eb0eed47a"
server_key_version: "encryptionkey@0"
I20260925 10:32:34.577122 4711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:34.597335 4711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:34.601256 4711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:34.601512 4711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:34.602430 4711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:34.603649 4711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:34.603756 4711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:34.603858 4711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:34.603979 4711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:34.635311 4711 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44167
I20260925 10:32:34.635358 4837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44167 every 8 connection(s)
I20260925 10:32:34.636143 4711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:32:34.639576 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 4711
I20260925 10:32:34.639763 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:32:34.640297 388 external_mini_cluster.cc:1428] Setting key 4df3d628cfd20d8994037324fc9e050d
Sep 25 10:32:34 dist-test-slave-dxdd krb5kdc[4472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:34.656386 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:34.638409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41663 (local address 127.0.97.62:42345)
0925 10:32:34.638586 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:34.638591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:34.639688 (+ 1097us) server_negotiation.cc:407] Connection header received
0925 10:32:34.641615 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:34.641620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:34.641721 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:34.641816 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:34.643473 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:34.644611 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:34.646102 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:34.646834 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:34.646978 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:34.650972 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:34.650977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:34.650980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:34.651011 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:34.653502 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:34.654195 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:34.654201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:34.654204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:34.654393 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:34.654793 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:34.654798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:34.654801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:34.655054 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:34.655202 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:34.655854 (+ 652us) server_negotiation.cc:299] Negotiation successful
0925 10:32:34.656063 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 10:32:34.657289 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:34.638865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42345 (local address 127.0.97.2:41663)
0925 10:32:34.639536 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:34.639576 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:34.641372 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:34.642085 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:34.642093 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:34.642703 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:34.643303 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:34.643319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:34.644773 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:34.644777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:34.645966 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:34.645977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:34.647021 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:34.647025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:34.647195 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:34.647923 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:34.647956 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:34.650745 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:34.653654 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:34.653662 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:34.653677 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:34.654071 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:34.654498 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:34.654504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:34.654507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:34.654677 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:34.655176 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:34.655184 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:34.655507 (+ 323us) client_negotiation.cc:769] Sending connection context
0925 10:32:34.655752 (+ 245us) client_negotiation.cc:240] Negotiation successful
0925 10:32:34.656240 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":298,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260925 10:32:34.658720 4838 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42345
I20260925 10:32:34.659103 4838 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:34.659700 4838 heartbeater.cc:511] Master 127.0.97.62:42345 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260925 10:32:34.661173 4516 ts_manager.cc:194] Registered new tserver with Master: b0d45977c89745729774336f604f617b (127.0.97.2:44167)
I20260925 10:32:34.661960 4516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:32:34.681717 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42345
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:34.990511 4843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:34.990762 4843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:34.990854 4843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:34.997778 4843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:34.997869 4843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:34.997980 4843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:32:35.007346 4843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:35.008479 4843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:35.009654 4843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:35.022013 4848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:35.021979 4851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:35.021930 4849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:35.022746 4843 server_base.cc:1034] running on GCE node
I20260925 10:32:35.023232 4843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:35.024034 4843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:35.025254 4843 hybrid_clock.cc:648] HybridClock initialized: now 1790332355025203 us; error 57 us; skew 500 ppm
I20260925 10:32:35.025552 4843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:35 dist-test-slave-dxdd krb5kdc[4472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:35.034590 4843 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:32:35.035712 4843 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:35.035836 4843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:35.036129 4843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:35.038895 4843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "1fcaf7853ec54885a3c5f747a4908107"
format_stamp: "Formatted at 2026-09-25 10:32:35 on dist-test-slave-dxdd"
server_key: "f6e17ad51d65249b09da4572ad245954"
server_key_iv: "e1f161c69960e98951bd45abee22aab4"
server_key_version: "encryptionkey@0"
I20260925 10:32:35.039674 4843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1fcaf7853ec54885a3c5f747a4908107"
format_stamp: "Formatted at 2026-09-25 10:32:35 on dist-test-slave-dxdd"
server_key: "f6e17ad51d65249b09da4572ad245954"
server_key_iv: "e1f161c69960e98951bd45abee22aab4"
server_key_version: "encryptionkey@0"
I20260925 10:32:35.044371 4843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:32:35.048521 4857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:35.050427 4843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:32:35.050594 4843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "1fcaf7853ec54885a3c5f747a4908107"
format_stamp: "Formatted at 2026-09-25 10:32:35 on dist-test-slave-dxdd"
server_key: "f6e17ad51d65249b09da4572ad245954"
server_key_iv: "e1f161c69960e98951bd45abee22aab4"
server_key_version: "encryptionkey@0"
I20260925 10:32:35.050884 4843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:35.077886 4843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:35.081809 4843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:35.082062 4843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:35.083026 4843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:35.084229 4843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:35.084342 4843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:35.084467 4843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:35.084558 4843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:35.115859 4843 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39691
I20260925 10:32:35.115907 4969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39691 every 8 connection(s)
I20260925 10:32:35.116631 4843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:32:35.117449 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 4843
I20260925 10:32:35.117626 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:32:35.118021 388 external_mini_cluster.cc:1428] Setting key dccb50ff374f0eb123f06f58870e737e
Sep 25 10:32:35 dist-test-slave-dxdd krb5kdc[4472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:35.135476 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.119008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57233 (local address 127.0.97.62:42345)
0925 10:32:35.119194 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:35.119200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:35.120501 (+ 1301us) server_negotiation.cc:407] Connection header received
0925 10:32:35.122482 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:35.122489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:35.122589 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:35.122694 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:35.124038 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.124769 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.126163 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.126598 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.126686 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.129889 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:35.129894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:35.129897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:35.129926 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:35.133073 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:35.133662 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:35.133666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:35.133668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:35.133772 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:35.134103 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:35.134106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:35.134108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:35.134445 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:35.134551 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:35.135110 (+ 559us) server_negotiation.cc:299] Negotiation successful
0925 10:32:35.135249 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:32:35.136301 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.119619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42345 (local address 127.0.97.3:57233)
0925 10:32:35.120366 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:35.120402 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:35.122171 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:35.122897 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:35.122904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:35.123388 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:35.123904 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:35.123919 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.124886 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.124889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:35.126039 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:35.126056 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.126766 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.126770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:35.126954 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.127633 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:35.127655 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:35.129698 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:35.133181 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:35.133187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:35.133197 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:35.133569 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:35.133858 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:35.133861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:35.133863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:35.134019 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:35.134545 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:35.134550 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:35.134817 (+ 267us) client_negotiation.cc:769] Sending connection context
0925 10:32:35.135041 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 10:32:35.135348 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":325,"threads_started":1}
I20260925 10:32:35.137507 4970 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42345
I20260925 10:32:35.137807 4970 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:35.138427 4970 heartbeater.cc:511] Master 127.0.97.62:42345 requested a full tablet report, sending...
I20260925 10:32:35.139715 4516 ts_manager.cc:194] Registered new tserver with Master: 1fcaf7853ec54885a3c5f747a4908107 (127.0.97.3:39691)
I20260925 10:32:35.140502 4516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57233
I20260925 10:32:35.144721 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:32:35.159816 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.148891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49306 (local address 127.0.97.62:42345)
0925 10:32:35.149078 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:35.149082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:35.149511 (+ 429us) server_negotiation.cc:407] Connection header received
0925 10:32:35.149653 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:35.149661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:35.149727 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:35.149813 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:35.151151 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.151796 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.152925 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.153350 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.153436 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.155535 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:35.155539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:35.155542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:35.155570 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:35.157720 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:35.158343 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:35.158348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:35.158350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:35.158452 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:35.158741 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:35.158744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:35.158746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:35.159009 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:35.159195 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:35.159470 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 10:32:35.159613 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 10:32:35.163472 4516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:32:35.171159 4516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:32:35.192849 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.182878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.0.97.3:39691)
0925 10:32:35.184612 (+ 1734us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:35.184623 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:35.184656 (+ 33us) server_negotiation.cc:407] Connection header received
0925 10:32:35.184749 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:35.184778 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:35.185045 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:35.185262 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:35.186416 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.188002 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.190266 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.191719 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.191851 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.192088 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:35.192351 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 10:32:35.192503 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1556,"thread_start_us":1375,"threads_started":1}
I20260925 10:32:35.192974 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.182621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39691 (local address 127.0.0.1:40618)
0925 10:32:35.183649 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:35.183702 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:35.183903 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:35.185572 (+ 1669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:35.185579 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:35.185647 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:35.186197 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:35.186213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.188216 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.188220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:35.190104 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:35.190120 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.191916 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.191923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:35.192114 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.192177 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 10:32:35.192312 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 10:32:35.192483 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":132,"spinlock_wait_cycles":3456,"thread_start_us":1935,"threads_started":1}
I20260925 10:32:35.193017 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.182856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43873 (local address 127.0.0.1:54412)
0925 10:32:35.184796 (+ 1940us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:35.184814 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:35.184924 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:35.185573 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:35.185582 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:35.185628 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:35.186036 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:35.186045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.188161 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.188169 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:35.190306 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:35.190339 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.192151 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.192157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:35.192329 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.192361 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:32:35.192422 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 10:32:35.192541 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1817,"mutex_wait_us":75,"spinlock_wait_cycles":4608,"thread_start_us":409,"threads_started":1}
I20260925 10:32:35.194193 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.183423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.0.97.1:43873)
0925 10:32:35.184316 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:35.184325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:35.185031 (+ 706us) server_negotiation.cc:407] Connection header received
0925 10:32:35.185125 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:35.185132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:35.185364 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:35.186177 (+ 813us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:35.186394 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.188014 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.190473 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.191905 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.193455 (+ 1550us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.193675 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:35.193737 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 10:32:35.193899 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":596,"threads_started":1}
I20260925 10:32:35.195266 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.185332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.0.97.2:44167)
0925 10:32:35.186800 (+ 1468us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:35.186813 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:35.186833 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:32:35.186897 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:35.186902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:35.187085 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:35.187290 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:35.188113 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.189675 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.192132 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.193546 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.193667 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.193831 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:35.194074 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 10:32:35.194227 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1326,"thread_start_us":1220,"threads_started":1}
I20260925 10:32:35.197034 4640 tablet_service.cc:1467] Processing CreateTablet for tablet de790f8ea6f8489ba1eaa07d626f9601 (DEFAULT_TABLE table=test-table [id=5577820cd709444e86598e764a7328d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:35.196973 4905 tablet_service.cc:1467] Processing CreateTablet for tablet de790f8ea6f8489ba1eaa07d626f9601 (DEFAULT_TABLE table=test-table [id=5577820cd709444e86598e764a7328d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:35.198021 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de790f8ea6f8489ba1eaa07d626f9601. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:35.198022 4905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de790f8ea6f8489ba1eaa07d626f9601. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:35.198766 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.184931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44167 (local address 127.0.0.1:56382)
0925 10:32:35.185516 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:35.185535 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:35.185661 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:35.187531 (+ 1870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:35.187535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:35.187554 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:35.187933 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:35.187945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.189813 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.189818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:35.191960 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:35.191980 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.193743 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.193749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:35.193904 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.193933 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 10:32:35.194022 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:32:35.194145 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"spinlock_wait_cycles":76672,"thread_start_us":560,"threads_started":1}
I20260925 10:32:35.201848 4705 heartbeater.cc:503] Master 127.0.97.62:42345 was elected leader, sending a full tablet report...
I20260925 10:32:35.202206 4773 tablet_service.cc:1467] Processing CreateTablet for tablet de790f8ea6f8489ba1eaa07d626f9601 (DEFAULT_TABLE table=test-table [id=5577820cd709444e86598e764a7328d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:35.203548 4773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de790f8ea6f8489ba1eaa07d626f9601. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:35.210212 4988 tablet_bootstrap.cc:492] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107: Bootstrap starting.
I20260925 10:32:35.213549 4988 tablet_bootstrap.cc:654] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:35.213886 4989 tablet_bootstrap.cc:492] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93: Bootstrap starting.
I20260925 10:32:35.214956 4988 log.cc:824] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:35.217509 4989 tablet_bootstrap.cc:654] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:35.217764 4991 tablet_bootstrap.cc:492] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b: Bootstrap starting.
I20260925 10:32:35.218679 4989 log.cc:824] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:35.218755 4988 tablet_bootstrap.cc:492] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107: No bootstrap required, opened a new log
I20260925 10:32:35.218976 4988 ts_tablet_manager.cc:1397] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 10:32:35.220438 4991 tablet_bootstrap.cc:654] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:35.221383 4989 tablet_bootstrap.cc:492] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93: No bootstrap required, opened a new log
I20260925 10:32:35.221552 4991 log.cc:824] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:35.221630 4989 ts_tablet_manager.cc:1397] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 10:32:35.223812 4991 tablet_bootstrap.cc:492] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b: No bootstrap required, opened a new log
I20260925 10:32:35.224049 4991 ts_tablet_manager.cc:1397] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 10:32:35.229147 4988 raft_consensus.cc:348] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcaf7853ec54885a3c5f747a4908107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39691 } } peers { permanent_uuid: "8b59ac1d2ca74bd8819f00975c903b93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43873 } } peers { permanent_uuid: "b0d45977c89745729774336f604f617b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44167 } }
I20260925 10:32:35.229599 4988 raft_consensus.cc:374] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:35.229696 4988 raft_consensus.cc:729] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcaf7853ec54885a3c5f747a4908107, State: Initialized, Role: FOLLOWER
I20260925 10:32:35.230283 4988 consensus_queue.cc:260] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcaf7853ec54885a3c5f747a4908107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39691 } } peers { permanent_uuid: "8b59ac1d2ca74bd8819f00975c903b93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43873 } } peers { permanent_uuid: "b0d45977c89745729774336f604f617b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44167 } }
I20260925 10:32:35.231410 4970 heartbeater.cc:503] Master 127.0.97.62:42345 was elected leader, sending a full tablet report...
I20260925 10:32:35.231376 4991 raft_consensus.cc:348] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcaf7853ec54885a3c5f747a4908107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39691 } } peers { permanent_uuid: "8b59ac1d2ca74bd8819f00975c903b93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43873 } } peers { permanent_uuid: "b0d45977c89745729774336f604f617b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44167 } }
I20260925 10:32:35.231741 4991 raft_consensus.cc:374] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:35.231830 4991 raft_consensus.cc:729] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d45977c89745729774336f604f617b, State: Initialized, Role: FOLLOWER
I20260925 10:32:35.231808 4989 raft_consensus.cc:348] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcaf7853ec54885a3c5f747a4908107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39691 } } peers { permanent_uuid: "8b59ac1d2ca74bd8819f00975c903b93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43873 } } peers { permanent_uuid: "b0d45977c89745729774336f604f617b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44167 } }
I20260925 10:32:35.232169 4989 raft_consensus.cc:374] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:35.232246 4989 raft_consensus.cc:729] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b59ac1d2ca74bd8819f00975c903b93, State: Initialized, Role: FOLLOWER
I20260925 10:32:35.232398 4991 consensus_queue.cc:260] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcaf7853ec54885a3c5f747a4908107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39691 } } peers { permanent_uuid: "8b59ac1d2ca74bd8819f00975c903b93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43873 } } peers { permanent_uuid: "b0d45977c89745729774336f604f617b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44167 } }
I20260925 10:32:35.232728 4989 consensus_queue.cc:260] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcaf7853ec54885a3c5f747a4908107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39691 } } peers { permanent_uuid: "8b59ac1d2ca74bd8819f00975c903b93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43873 } } peers { permanent_uuid: "b0d45977c89745729774336f604f617b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44167 } }
I20260925 10:32:35.233915 4988 ts_tablet_manager.cc:1428] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260925 10:32:35.235337 4989 ts_tablet_manager.cc:1428] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93: Time spent starting tablet: real 0.014s user 0.004s sys 0.010s
I20260925 10:32:35.235881 4994 raft_consensus.cc:482] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:32:35.236126 4994 raft_consensus.cc:504] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcaf7853ec54885a3c5f747a4908107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39691 } } peers { permanent_uuid: "8b59ac1d2ca74bd8819f00975c903b93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43873 } } peers { permanent_uuid: "b0d45977c89745729774336f604f617b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44167 } }
I20260925 10:32:35.239524 4838 heartbeater.cc:503] Master 127.0.97.62:42345 was elected leader, sending a full tablet report...
I20260925 10:32:35.240200 4991 ts_tablet_manager.cc:1428] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260925 10:32:35.240880 4994 leader_election.cc:288] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b59ac1d2ca74bd8819f00975c903b93 (127.0.97.1:43873), b0d45977c89745729774336f604f617b (127.0.97.2:44167)
I20260925 10:32:35.246650 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.239603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43873 (local address 127.0.97.3:36621)
0925 10:32:35.239854 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:35.239875 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:35.239977 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:35.241074 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:35.241079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:35.241102 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:35.241274 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:35.241283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.242734 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.242738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:35.244593 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:35.244609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.246021 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.246027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:35.246178 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.246201 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:32:35.246269 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 10:32:35.246433 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":26}
I20260925 10:32:35.246702 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.240582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36621 (local address 127.0.97.1:43873)
0925 10:32:35.240750 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:35.240755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:35.240772 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:32:35.240846 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:35.240853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:35.240927 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:35.241029 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:35.241407 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.242575 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.244762 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.245847 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.246014 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.246117 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:35.246296 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 10:32:35.246464 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:32:35.247771 4660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de790f8ea6f8489ba1eaa07d626f9601" candidate_uuid: "1fcaf7853ec54885a3c5f747a4908107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b59ac1d2ca74bd8819f00975c903b93" is_pre_election: true
I20260925 10:32:35.248096 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.241281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44167 (local address 127.0.97.3:47111)
0925 10:32:35.241949 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:35.241966 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:35.242125 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:35.242521 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:35.242527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:35.242551 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:35.242976 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:35.242987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.244290 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.244295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:35.245910 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:35.245921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.247526 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.247531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:35.247641 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.247663 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 10:32:35.247735 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 10:32:35.247813 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":3,"thread_start_us":556,"threads_started":1}
I20260925 10:32:35.248163 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.241663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47111 (local address 127.0.97.2:44167)
0925 10:32:35.241834 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:35.241839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:35.242039 (+ 200us) server_negotiation.cc:407] Connection header received
0925 10:32:35.242259 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:35.242264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:35.242359 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:35.242461 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:35.243167 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.244162 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.246080 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.247280 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.247420 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.247509 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:35.247790 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 10:32:35.247916 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:32:35.248303 4660 raft_consensus.cc:2454] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fcaf7853ec54885a3c5f747a4908107 in term 0.
I20260925 10:32:35.248960 4861 leader_election.cc:302] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcaf7853ec54885a3c5f747a4908107, 8b59ac1d2ca74bd8819f00975c903b93; no voters:
I20260925 10:32:35.249018 4793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de790f8ea6f8489ba1eaa07d626f9601" candidate_uuid: "1fcaf7853ec54885a3c5f747a4908107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d45977c89745729774336f604f617b" is_pre_election: true
I20260925 10:32:35.249316 4994 raft_consensus.cc:2781] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:32:35.249468 4994 raft_consensus.cc:482] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:32:35.249603 4994 raft_consensus.cc:3037] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:35.249577 4793 raft_consensus.cc:2454] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fcaf7853ec54885a3c5f747a4908107 in term 0.
I20260925 10:32:35.251863 4994 raft_consensus.cc:504] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcaf7853ec54885a3c5f747a4908107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39691 } } peers { permanent_uuid: "8b59ac1d2ca74bd8819f00975c903b93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43873 } } peers { permanent_uuid: "b0d45977c89745729774336f604f617b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44167 } }
I20260925 10:32:35.252393 4994 leader_election.cc:288] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [CANDIDATE]: Term 1 election: Requested vote from peers 8b59ac1d2ca74bd8819f00975c903b93 (127.0.97.1:43873), b0d45977c89745729774336f604f617b (127.0.97.2:44167)
I20260925 10:32:35.252750 4660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de790f8ea6f8489ba1eaa07d626f9601" candidate_uuid: "1fcaf7853ec54885a3c5f747a4908107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b59ac1d2ca74bd8819f00975c903b93"
I20260925 10:32:35.252796 4793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de790f8ea6f8489ba1eaa07d626f9601" candidate_uuid: "1fcaf7853ec54885a3c5f747a4908107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d45977c89745729774336f604f617b"
I20260925 10:32:35.252950 4660 raft_consensus.cc:3037] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:35.252960 4793 raft_consensus.cc:3037] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:35.255224 4793 raft_consensus.cc:2454] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcaf7853ec54885a3c5f747a4908107 in term 1.
I20260925 10:32:35.255234 4660 raft_consensus.cc:2454] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcaf7853ec54885a3c5f747a4908107 in term 1.
I20260925 10:32:35.255580 4860 leader_election.cc:302] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcaf7853ec54885a3c5f747a4908107, b0d45977c89745729774336f604f617b; no voters:
I20260925 10:32:35.255870 4994 raft_consensus.cc:2781] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:35.256296 4994 raft_consensus.cc:686] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [term 1 LEADER]: Becoming Leader. State: Replica: 1fcaf7853ec54885a3c5f747a4908107, State: Running, Role: LEADER
I20260925 10:32:35.256763 4994 consensus_queue.cc:237] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcaf7853ec54885a3c5f747a4908107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39691 } } peers { permanent_uuid: "8b59ac1d2ca74bd8819f00975c903b93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43873 } } peers { permanent_uuid: "b0d45977c89745729774336f604f617b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44167 } }
I20260925 10:32:35.260797 4515 catalog_manager.cc:5495] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fcaf7853ec54885a3c5f747a4908107 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1fcaf7853ec54885a3c5f747a4908107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcaf7853ec54885a3c5f747a4908107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b59ac1d2ca74bd8819f00975c903b93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0d45977c89745729774336f604f617b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44167 } health_report { overall_health: UNKNOWN } } }
I20260925 10:32:35.293141 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.287866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40626 (local address 127.0.97.3:39691)
0925 10:32:35.288057 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:35.288063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:35.288159 (+ 96us) server_negotiation.cc:407] Connection header received
0925 10:32:35.288391 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:35.288395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:35.288472 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:35.288646 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:32:35.289026 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.289758 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.291145 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.291528 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.291652 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.292109 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:32:35.292603 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:32:35.292754 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:35.292802 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 10:32:35.292874 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:32:35.300824 4660 raft_consensus.cc:1261] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93 [term 1 FOLLOWER]: Refusing update from remote peer 1fcaf7853ec54885a3c5f747a4908107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:35.300824 4793 raft_consensus.cc:1261] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b [term 1 FOLLOWER]: Refusing update from remote peer 1fcaf7853ec54885a3c5f747a4908107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:35.301802 4998 consensus_queue.cc:1035] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d45977c89745729774336f604f617b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:35.302160 4994 consensus_queue.cc:1035] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b59ac1d2ca74bd8819f00975c903b93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:35.313352 5003 mvcc.cc:205] Tried to move back new op lower bound from 7333201327302623232 to 7333201327135244288. Current Snapshot: MvccSnapshot[applied={T|T < 7333201327302623232}]
I20260925 10:32:35.313421 5001 mvcc.cc:205] Tried to move back new op lower bound from 7333201327302623232 to 7333201327135244288. Current Snapshot: MvccSnapshot[applied={T|T < 7333201327302623232}]
I20260925 10:32:35.330705 4515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:35.330986 4515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:35.334504 4515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fcdf46fd3d4462a902f2294cf306bc1: Sending DeleteTablet for 3 replicas of tablet de790f8ea6f8489ba1eaa07d626f9601
I20260925 10:32:35.335510 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet de790f8ea6f8489ba1eaa07d626f9601 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54412
I20260925 10:32:35.335796 4905 tablet_service.cc:1514] Processing DeleteTablet for tablet de790f8ea6f8489ba1eaa07d626f9601 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40618
I20260925 10:32:35.336377 4773 tablet_service.cc:1514] Processing DeleteTablet for tablet de790f8ea6f8489ba1eaa07d626f9601 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56382
I20260925 10:32:35.336755 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 4578
I20260925 10:32:35.351306 5011 tablet_replica.cc:330] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93: stopping tablet replica
I20260925 10:32:35.351771 5011 raft_consensus.cc:2229] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:35.351850 5013 tablet_replica.cc:330] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b: stopping tablet replica
I20260925 10:32:35.352191 5013 raft_consensus.cc:2229] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:35.352276 5011 raft_consensus.cc:2258] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:35.352682 5013 raft_consensus.cc:2258] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:35.354264 5011 ts_tablet_manager.cc:1905] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:35.354941 5012 tablet_replica.cc:330] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107: stopping tablet replica
I20260925 10:32:35.355394 5012 raft_consensus.cc:2229] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:32:35.356513 5012 raft_consensus.cc:2258] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:35.357249 5013 ts_tablet_manager.cc:1905] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:35.358234 5012 ts_tablet_manager.cc:1905] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:35.361305 5011 ts_tablet_manager.cc:1918] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:32:35.361488 5011 log.cc:1196] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/de790f8ea6f8489ba1eaa07d626f9601
I20260925 10:32:35.362046 5011 ts_tablet_manager.cc:1939] T de790f8ea6f8489ba1eaa07d626f9601 P 8b59ac1d2ca74bd8819f00975c903b93: Deleting consensus metadata
I20260925 10:32:35.363865 4503 catalog_manager.cc:4841] TS 8b59ac1d2ca74bd8819f00975c903b93 (127.0.97.1:43873): tablet de790f8ea6f8489ba1eaa07d626f9601 (table test-table [id=5577820cd709444e86598e764a7328d3]) successfully deleted
I20260925 10:32:35.365222 5012 ts_tablet_manager.cc:1918] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:32:35.365373 5012 log.cc:1196] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/de790f8ea6f8489ba1eaa07d626f9601
I20260925 10:32:35.365927 5012 ts_tablet_manager.cc:1939] T de790f8ea6f8489ba1eaa07d626f9601 P 1fcaf7853ec54885a3c5f747a4908107: Deleting consensus metadata
I20260925 10:32:35.367020 5013 ts_tablet_manager.cc:1918] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:32:35.367242 5013 log.cc:1196] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/de790f8ea6f8489ba1eaa07d626f9601
Sep 25 10:32:35 dist-test-slave-dxdd krb5kdc[4472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:32:35.367449 4502 catalog_manager.cc:4841] TS 1fcaf7853ec54885a3c5f747a4908107 (127.0.97.3:39691): tablet de790f8ea6f8489ba1eaa07d626f9601 (table test-table [id=5577820cd709444e86598e764a7328d3]) successfully deleted
I20260925 10:32:35.367918 5013 ts_tablet_manager.cc:1939] T de790f8ea6f8489ba1eaa07d626f9601 P b0d45977c89745729774336f604f617b: Deleting consensus metadata
I20260925 10:32:35.369334 4502 catalog_manager.cc:4841] TS b0d45977c89745729774336f604f617b (127.0.97.2:44167): tablet de790f8ea6f8489ba1eaa07d626f9601 (table test-table [id=5577820cd709444e86598e764a7328d3]) successfully deleted
I20260925 10:32:35.374818 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.338984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.0.97.1:43873)
0925 10:32:35.339201 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:35.339209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:35.348746 (+ 9537us) server_negotiation.cc:407] Connection header received
0925 10:32:35.349280 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:35.349289 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:35.349394 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:35.349503 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:35.353118 (+ 3615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.354226 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.363141 (+ 8915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.363773 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.363871 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.368646 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:35.368658 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:35.368674 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:35.368714 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:35.372197 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:35.372821 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:35.372830 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:35.372833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:35.372982 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:35.373307 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:35.373312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:35.373316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:35.373766 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:35.374053 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:35.374208 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 10:32:35.374438 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:32:35.375387 4700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:35.507633 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 4578
I20260925 10:32:35.524500 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 4711
Sep 25 10:32:35 dist-test-slave-dxdd krb5kdc[4472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:32:35.539227 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.525241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56386 (local address 127.0.97.2:44167)
0925 10:32:35.525425 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:35.525433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:35.525515 (+ 82us) server_negotiation.cc:407] Connection header received
0925 10:32:35.525716 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:35.525722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:35.525822 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:35.525926 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:35.527153 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.527949 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.529280 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.529694 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.529820 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.533912 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:35.533931 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:35.533952 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:35.533989 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:35.536663 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:35.537280 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:35.537290 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:35.537293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:35.537424 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:35.537780 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:35.537785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:35.537788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:35.538128 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:35.538547 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:35.538718 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 10:32:35.538919 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:32:35.539825 4833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:35.644515 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 4711
I20260925 10:32:35.661490 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 4843
Sep 25 10:32:35 dist-test-slave-dxdd krb5kdc[4472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:32:35.678181 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:35.662279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40636 (local address 127.0.97.3:39691)
0925 10:32:35.662446 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:35.662450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:35.662517 (+ 67us) server_negotiation.cc:407] Connection header received
0925 10:32:35.662708 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:35.662712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:35.662772 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:35.662864 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:35.664609 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.665389 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.667131 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:35.667541 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:35.667630 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:35.672487 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:35.672500 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:35.672517 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:35.672554 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:35.675537 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:35.676187 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:35.676200 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:35.676202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:35.676355 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:35.676722 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:35.676726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:35.676728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:35.677047 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:35.677340 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:35.677612 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 10:32:35.677843 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 10:32:35.678939 4965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:35.798226 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 4843
I20260925 10:32:35.817000 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 4485
I20260925 10:32:35.818048 4546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:35.922151 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 4485
2026-09-25T10:32:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3999 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T10:32:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:32:35Z Disabled control of system clock
I20260925 10:32:35.987030 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42465
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 10:32:36.297012 5022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:36.297268 5022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:36.297350 5022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:36.304380 5022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:36.304476 5022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:36.304512 5022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:36.304602 5022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:36.304648 5022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:32:36.313421 5022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42465
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:36.314608 5022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:36.316211 5022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:36.327477 5027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:36.327477 5028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:36.327517 5030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:36.329023 5022 server_base.cc:1034] running on GCE node
I20260925 10:32:36.329864 5022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:36.331314 5022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:36.332515 5022 hybrid_clock.cc:648] HybridClock initialized: now 1790332356332461 us; error 52 us; skew 500 ppm
I20260925 10:32:36.332885 5022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:32:36.335162 5022 webserver.cc:460] Webserver started at http://127.0.97.62:43367/ using document root <none> and password file <none>
I20260925 10:32:36.335817 5022 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:36.335928 5022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:36.336274 5022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:36.339219 5022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "203955ffa5ff4cc49873203ae0832dea"
format_stamp: "Formatted at 2026-09-25 10:32:36 on dist-test-slave-dxdd"
server_key: "b2dbbcb875ae68e46a027ba7bab179a8"
server_key_iv: "b26ca92159738e6e06f62bff04977b54"
server_key_version: "encryptionkey@0"
I20260925 10:32:36.340006 5022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "203955ffa5ff4cc49873203ae0832dea"
format_stamp: "Formatted at 2026-09-25 10:32:36 on dist-test-slave-dxdd"
server_key: "b2dbbcb875ae68e46a027ba7bab179a8"
server_key_iv: "b26ca92159738e6e06f62bff04977b54"
server_key_version: "encryptionkey@0"
I20260925 10:32:36.344975 5022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 10:32:36.348452 5036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:36.349797 5022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:32:36.349972 5022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "203955ffa5ff4cc49873203ae0832dea"
format_stamp: "Formatted at 2026-09-25 10:32:36 on dist-test-slave-dxdd"
server_key: "b2dbbcb875ae68e46a027ba7bab179a8"
server_key_iv: "b26ca92159738e6e06f62bff04977b54"
server_key_version: "encryptionkey@0"
I20260925 10:32:36.350204 5022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:36.355917 5022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:36.356698 5022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:36.357007 5022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:36.375514 5022 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42465
I20260925 10:32:36.375543 5087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42465 every 8 connection(s)
I20260925 10:32:36.376853 5022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:32:36.380708 5088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:36.382810 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 5022
I20260925 10:32:36.382997 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:32:36.383792 388 external_mini_cluster.cc:1428] Setting key 98f196925f8442ce4028518d909b5382
I20260925 10:32:36.392094 5091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:36.385883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41592 (local address 127.0.97.62:42465)
0925 10:32:36.386976 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:36.386987 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:36.387029 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:32:36.388688 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:36.388710 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:36.389127 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:36.389464 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:36.390540 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:36.390546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:36.390559 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:36.390572 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:36.390640 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:36.390656 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:36.390831 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:36.390935 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 10:32:36.391199 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":448,"threads_started":1}
I20260925 10:32:36.392621 5088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea: Bootstrap starting.
I20260925 10:32:36.395462 5088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:36.396395 5088 log.cc:824] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:36.398866 5088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea: No bootstrap required, opened a new log
I20260925 10:32:36.405450 5088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203955ffa5ff4cc49873203ae0832dea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42465 } }
I20260925 10:32:36.405797 5088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:36.405922 5088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 203955ffa5ff4cc49873203ae0832dea, State: Initialized, Role: FOLLOWER
I20260925 10:32:36.406500 5088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203955ffa5ff4cc49873203ae0832dea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42465 } }
I20260925 10:32:36.406746 5088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:32:36.406876 5088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:32:36.407042 5088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:36.409024 5088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203955ffa5ff4cc49873203ae0832dea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42465 } }
I20260925 10:32:36.409480 5088 leader_election.cc:302] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 203955ffa5ff4cc49873203ae0832dea; no voters:
I20260925 10:32:36.410063 5088 leader_election.cc:288] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:32:36.410226 5093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:36.411012 5093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [term 1 LEADER]: Becoming Leader. State: Replica: 203955ffa5ff4cc49873203ae0832dea, State: Running, Role: LEADER
I20260925 10:32:36.411531 5093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203955ffa5ff4cc49873203ae0832dea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42465 } }
I20260925 10:32:36.412087 5088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:36.414285 5095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 203955ffa5ff4cc49873203ae0832dea. Latest consensus state: current_term: 1 leader_uuid: "203955ffa5ff4cc49873203ae0832dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203955ffa5ff4cc49873203ae0832dea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42465 } } }
I20260925 10:32:36.414372 5094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "203955ffa5ff4cc49873203ae0832dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203955ffa5ff4cc49873203ae0832dea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42465 } } }
I20260925 10:32:36.414722 5094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:36.414726 5095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:36.415742 5100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:36.421814 5100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:36.430238 5100 catalog_manager.cc:1347] Generated new cluster ID: e078339c65854ae5b445a5c3e5bb42d3
I20260925 10:32:36.430351 5100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:36.456178 5100 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:32:36.456683 5100 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:36.476155 5100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea: Generated new TSK 0
I20260925 10:32:36.477226 5100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:32:36.491488 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42465
--builtin_ntp_servers=127.0.97.20:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 10:32:36.806874 5112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:36.807132 5112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:36.807292 5112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:36.814365 5112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:36.814465 5112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:36.814651 5112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:32:36.823354 5112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:36.824595 5112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:36.826236 5112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:36.838778 5118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:36.839155 5120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:36.839223 5117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:36.843662 5112 server_base.cc:1034] running on GCE node
I20260925 10:32:36.844214 5112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:36.845099 5112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:36.846375 5112 hybrid_clock.cc:648] HybridClock initialized: now 1790332356846276 us; error 37 us; skew 500 ppm
I20260925 10:32:36.846662 5112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:32:36.848851 5112 webserver.cc:460] Webserver started at http://127.0.97.1:39667/ using document root <none> and password file <none>
I20260925 10:32:36.849485 5112 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:36.849628 5112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:36.849934 5112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:36.852734 5112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "b212fae77339447a96f16f629c26303d"
format_stamp: "Formatted at 2026-09-25 10:32:36 on dist-test-slave-dxdd"
server_key: "4deecdad3e02fd082bb107adecd0de6f"
server_key_iv: "b7d91f28df63ebaa8991442711b78809"
server_key_version: "encryptionkey@0"
I20260925 10:32:36.853505 5112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b212fae77339447a96f16f629c26303d"
format_stamp: "Formatted at 2026-09-25 10:32:36 on dist-test-slave-dxdd"
server_key: "4deecdad3e02fd082bb107adecd0de6f"
server_key_iv: "b7d91f28df63ebaa8991442711b78809"
server_key_version: "encryptionkey@0"
I20260925 10:32:36.858426 5112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 10:32:36.861521 5126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:36.862859 5112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 10:32:36.863016 5112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "b212fae77339447a96f16f629c26303d"
format_stamp: "Formatted at 2026-09-25 10:32:36 on dist-test-slave-dxdd"
server_key: "4deecdad3e02fd082bb107adecd0de6f"
server_key_iv: "b7d91f28df63ebaa8991442711b78809"
server_key_version: "encryptionkey@0"
I20260925 10:32:36.863229 5112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:36.868286 5112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:36.869002 5112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:36.869247 5112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:36.870100 5112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:36.871384 5112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:36.871505 5112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:36.871618 5112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:36.871698 5112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:36.903283 5112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46749
I20260925 10:32:36.903365 5238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46749 every 8 connection(s)
I20260925 10:32:36.904510 5112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:32:36.908455 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 5112
I20260925 10:32:36.908691 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:32:36.909307 388 external_mini_cluster.cc:1428] Setting key 67c4e7871428d722019b2d87c6faf445
I20260925 10:32:36.911859 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42465
--builtin_ntp_servers=127.0.97.20:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 10:32:36.912525 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:36.906899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52569 (local address 127.0.97.62:42465)
0925 10:32:36.907292 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:36.907297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:36.908427 (+ 1130us) server_negotiation.cc:407] Connection header received
0925 10:32:36.910218 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:36.910223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:36.910296 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:36.910455 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:36.911725 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:36.911731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:36.911736 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:36.911753 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:36.911804 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:36.911816 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:36.911931 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:36.912188 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 10:32:36.912250 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":209,"threads_started":1}
I20260925 10:32:36.913424 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:36.907470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42465 (local address 127.0.97.1:52569)
0925 10:32:36.908278 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:36.908313 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:36.909972 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:36.910676 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:36.910684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:36.910768 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:36.911458 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:32:36.911461 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:36.911496 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:32:36.911570 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:36.911918 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:36.911928 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:36.911936 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 10:32:36.912258 (+ 322us) client_negotiation.cc:240] Negotiation successful
0925 10:32:36.912487 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":241,"threads_started":1}
I20260925 10:32:36.914906 5239 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42465
I20260925 10:32:36.915135 5239 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:36.915673 5239 heartbeater.cc:511] Master 127.0.97.62:42465 requested a full tablet report, sending...
I20260925 10:32:36.917668 5053 ts_manager.cc:194] Registered new tserver with Master: b212fae77339447a96f16f629c26303d (127.0.97.1:46749)
W20260925 10:32:37.223389 5243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:37.223645 5243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:37.223737 5243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:37.230650 5243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:37.230741 5243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:37.230841 5243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:32:37.239274 5243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:37.240501 5243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:37.242141 5243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:37.254733 5249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:37.254858 5248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:37.254786 5251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:37.255712 5243 server_base.cc:1034] running on GCE node
I20260925 10:32:37.256328 5243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:37.257177 5243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:37.258410 5243 hybrid_clock.cc:648] HybridClock initialized: now 1790332357258366 us; error 48 us; skew 500 ppm
I20260925 10:32:37.258705 5243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:32:37.260883 5243 webserver.cc:460] Webserver started at http://127.0.97.2:33315/ using document root <none> and password file <none>
I20260925 10:32:37.261508 5243 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:37.261627 5243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:37.261925 5243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:37.264607 5243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "4c6683de89f54a10a726e64c7140b40d"
format_stamp: "Formatted at 2026-09-25 10:32:37 on dist-test-slave-dxdd"
server_key: "0308090ae000a8b34dfe8437bebd063e"
server_key_iv: "22d9cc8359829fc5517b62d3ccad0bc7"
server_key_version: "encryptionkey@0"
I20260925 10:32:37.265352 5243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4c6683de89f54a10a726e64c7140b40d"
format_stamp: "Formatted at 2026-09-25 10:32:37 on dist-test-slave-dxdd"
server_key: "0308090ae000a8b34dfe8437bebd063e"
server_key_iv: "22d9cc8359829fc5517b62d3ccad0bc7"
server_key_version: "encryptionkey@0"
I20260925 10:32:37.270131 5243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 10:32:37.273283 5257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:37.274537 5243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:32:37.274694 5243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "4c6683de89f54a10a726e64c7140b40d"
format_stamp: "Formatted at 2026-09-25 10:32:37 on dist-test-slave-dxdd"
server_key: "0308090ae000a8b34dfe8437bebd063e"
server_key_iv: "22d9cc8359829fc5517b62d3ccad0bc7"
server_key_version: "encryptionkey@0"
I20260925 10:32:37.274914 5243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:37.280056 5243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:37.280778 5243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:37.281033 5243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:37.281854 5243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:37.283200 5243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:37.283313 5243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:37.283448 5243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:37.283532 5243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:37.315227 5243 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39845
I20260925 10:32:37.315281 5369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39845 every 8 connection(s)
I20260925 10:32:37.316531 5243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:32:37.321424 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 5243
I20260925 10:32:37.321587 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:32:37.322120 388 external_mini_cluster.cc:1428] Setting key 29222320ca2a829967d4ae1d94972c14
I20260925 10:32:37.325044 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42465
--builtin_ntp_servers=127.0.97.20:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 10:32:37.325415 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.319179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52051 (local address 127.0.97.62:42465)
0925 10:32:37.319381 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:37.319387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:37.320742 (+ 1355us) server_negotiation.cc:407] Connection header received
0925 10:32:37.323042 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:37.323046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:37.323123 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:37.323240 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:37.324595 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:37.324601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:37.324606 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:37.324626 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:37.324677 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:37.324695 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:37.324825 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:37.325003 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 10:32:37.325064 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:32:37.326220 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.319750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42465 (local address 127.0.97.2:52051)
0925 10:32:37.320583 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:37.320623 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:37.322749 (+ 2126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:37.323484 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:37.323495 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:37.323606 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:37.324312 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:32:37.324315 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:37.324353 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:32:37.324432 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:37.324795 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:37.324801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:37.324806 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:32:37.325027 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 10:32:37.325269 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":247,"threads_started":1}
I20260925 10:32:37.327517 5370 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42465
I20260925 10:32:37.327814 5370 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:37.328409 5370 heartbeater.cc:511] Master 127.0.97.62:42465 requested a full tablet report, sending...
I20260925 10:32:37.329789 5053 ts_manager.cc:194] Registered new tserver with Master: 4c6683de89f54a10a726e64c7140b40d (127.0.97.2:39845)
W20260925 10:32:37.637952 5373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:37.638192 5373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:37.638274 5373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:37.645238 5373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:37.645334 5373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:37.645511 5373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:32:37.654633 5373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:37.655807 5373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:37.657529 5373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:37.670553 5379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:37.670717 5378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:37.670737 5381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:37.671499 5373 server_base.cc:1034] running on GCE node
I20260925 10:32:37.672016 5373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:37.672902 5373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:37.674114 5373 hybrid_clock.cc:648] HybridClock initialized: now 1790332357674062 us; error 45 us; skew 500 ppm
I20260925 10:32:37.674472 5373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:32:37.676476 5373 webserver.cc:460] Webserver started at http://127.0.97.3:42619/ using document root <none> and password file <none>
I20260925 10:32:37.677093 5373 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:37.677218 5373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:37.677517 5373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:37.680234 5373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "e062531cf4134c1aa298c25ce138f8f5"
format_stamp: "Formatted at 2026-09-25 10:32:37 on dist-test-slave-dxdd"
server_key: "98297dc81f2ab7a4faa450b58b1f9261"
server_key_iv: "c743e89e0feacaa1b34e0c64b01be9dc"
server_key_version: "encryptionkey@0"
I20260925 10:32:37.681044 5373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e062531cf4134c1aa298c25ce138f8f5"
format_stamp: "Formatted at 2026-09-25 10:32:37 on dist-test-slave-dxdd"
server_key: "98297dc81f2ab7a4faa450b58b1f9261"
server_key_iv: "c743e89e0feacaa1b34e0c64b01be9dc"
server_key_version: "encryptionkey@0"
I20260925 10:32:37.685633 5373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 10:32:37.688969 5387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:37.690171 5373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:32:37.690376 5373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "e062531cf4134c1aa298c25ce138f8f5"
format_stamp: "Formatted at 2026-09-25 10:32:37 on dist-test-slave-dxdd"
server_key: "98297dc81f2ab7a4faa450b58b1f9261"
server_key_iv: "c743e89e0feacaa1b34e0c64b01be9dc"
server_key_version: "encryptionkey@0"
I20260925 10:32:37.690586 5373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:37.695679 5373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:37.696401 5373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:37.696681 5373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:37.697525 5373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:37.698845 5373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:37.698958 5373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:37.699087 5373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:37.699177 5373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:37.730786 5373 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39227
I20260925 10:32:37.730840 5499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39227 every 8 connection(s)
I20260925 10:32:37.732003 5373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:32:37.733177 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 5373
I20260925 10:32:37.733335 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:32:37.733879 388 external_mini_cluster.cc:1428] Setting key b20357e235009d8ed08e7a9fa135b84b
I20260925 10:32:37.740199 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.734596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54257 (local address 127.0.97.62:42465)
0925 10:32:37.734795 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:37.734801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:37.736295 (+ 1494us) server_negotiation.cc:407] Connection header received
0925 10:32:37.738142 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:37.738148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:37.738208 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:37.738341 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:37.739494 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:37.739500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:37.739504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:37.739520 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:37.739557 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:37.739568 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:37.739681 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:37.739864 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 10:32:37.739931 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:32:37.741019 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.735171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42465 (local address 127.0.97.3:54257)
0925 10:32:37.736147 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:37.736185 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:37.737893 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:37.738566 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:37.738573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:37.738668 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:37.739217 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:32:37.739219 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:37.739251 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:32:37.739324 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:37.739664 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:37.739673 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:37.739680 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:32:37.739942 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 10:32:37.740179 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":289,"threads_started":1}
I20260925 10:32:37.742134 5500 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42465
I20260925 10:32:37.742444 5500 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:37.742944 5500 heartbeater.cc:511] Master 127.0.97.62:42465 requested a full tablet report, sending...
I20260925 10:32:37.744145 5053 ts_manager.cc:194] Registered new tserver with Master: e062531cf4134c1aa298c25ce138f8f5 (127.0.97.3:39227)
I20260925 10:32:37.748669 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:32:37.754858 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.752627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.0.97.62:42465)
0925 10:32:37.752819 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:37.752827 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:37.753206 (+ 379us) server_negotiation.cc:407] Connection header received
0925 10:32:37.753324 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:37.753327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:37.753368 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:37.753439 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:37.754257 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:37.754261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:37.754264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:37.754277 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:37.754350 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:37.754372 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:37.754460 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:37.754513 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 10:32:37.754551 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:32:37.758100 5053 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:32:37.764673 5053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:32:37.782027 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.775755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46749 (local address 127.0.0.1:41390)
0925 10:32:37.778543 (+ 2788us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:37.778556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:37.778676 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:37.779477 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:37.779485 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:37.779573 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:37.780449 (+ 876us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:32:37.780454 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:37.780487 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:32:37.780560 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:37.781274 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:37.781283 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:37.781290 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:32:37.781445 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 10:32:37.781529 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2670,"thread_start_us":4900,"threads_started":1}
I20260925 10:32:37.782958 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.776229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.0.97.1:46749)
0925 10:32:37.777716 (+ 1487us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:37.777727 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:37.778655 (+ 928us) server_negotiation.cc:407] Connection header received
0925 10:32:37.778800 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:37.778810 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:37.779090 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:37.779251 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:37.781029 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:37.781037 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:37.781051 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:37.781077 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:37.781132 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:37.781151 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:37.782436 (+ 1285us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:37.782512 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 10:32:37.782608 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1318,"thread_start_us":1173,"threads_started":1}
I20260925 10:32:37.783738 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.775526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39845 (local address 127.0.0.1:43602)
0925 10:32:37.776614 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:37.776651 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:37.776758 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:37.781299 (+ 4541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:37.781303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:37.781322 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:37.782116 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:32:37.782121 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:37.782140 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:32:37.782191 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:37.783139 (+ 948us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:37.783147 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:37.783152 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:32:37.783274 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 10:32:37.783324 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"thread_start_us":742,"threads_started":1}
I20260925 10:32:37.784135 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.778295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.0.97.3:39227)
0925 10:32:37.779350 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:37.779358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:37.781045 (+ 1687us) server_negotiation.cc:407] Connection header received
0925 10:32:37.781269 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:37.781277 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:37.781468 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:37.781619 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:37.783322 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:37.783327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:37.783336 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:37.783349 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:37.783389 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:37.783402 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:37.783589 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:37.783650 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 10:32:37.783733 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":620,"threads_started":1}
I20260925 10:32:37.784166 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.777589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39227 (local address 127.0.0.1:58996)
0925 10:32:37.780941 (+ 3352us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:37.780959 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:37.781131 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:37.782543 (+ 1412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:37.782547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:37.782568 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:37.783110 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:32:37.783113 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:37.783128 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:32:37.783168 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:37.783502 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:37.783506 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:37.783515 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:32:37.783652 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 10:32:37.783703 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3221,"thread_start_us":721,"threads_started":1}
I20260925 10:32:37.784696 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.775772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43602 (local address 127.0.97.2:39845)
0925 10:32:37.780882 (+ 5110us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:37.780891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:37.780917 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:32:37.780981 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:37.780987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:37.781176 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:37.781313 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:37.782603 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:37.782611 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:37.782622 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:37.782647 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:37.782699 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:37.782718 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:37.784207 (+ 1489us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:37.784278 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 10:32:37.784368 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4960,"thread_start_us":4509,"threads_started":1}
I20260925 10:32:37.787015 5174 tablet_service.cc:1467] Processing CreateTablet for tablet dbbe763ba3bf4fbfa5846b2115d2a7b0 (DEFAULT_TABLE table=test-table [id=a2bb0527fed848779863d1ceb942b309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:37.788183 5174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbe763ba3bf4fbfa5846b2115d2a7b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:37.787930 5305 tablet_service.cc:1467] Processing CreateTablet for tablet dbbe763ba3bf4fbfa5846b2115d2a7b0 (DEFAULT_TABLE table=test-table [id=a2bb0527fed848779863d1ceb942b309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:37.787917 5435 tablet_service.cc:1467] Processing CreateTablet for tablet dbbe763ba3bf4fbfa5846b2115d2a7b0 (DEFAULT_TABLE table=test-table [id=a2bb0527fed848779863d1ceb942b309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:37.789258 5435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbe763ba3bf4fbfa5846b2115d2a7b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:37.789259 5305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbe763ba3bf4fbfa5846b2115d2a7b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:37.799208 5518 tablet_bootstrap.cc:492] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d: Bootstrap starting.
I20260925 10:32:37.802672 5518 tablet_bootstrap.cc:654] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:37.803053 5519 tablet_bootstrap.cc:492] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5: Bootstrap starting.
I20260925 10:32:37.803854 5520 tablet_bootstrap.cc:492] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d: Bootstrap starting.
I20260925 10:32:37.804014 5518 log.cc:824] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:37.807381 5520 tablet_bootstrap.cc:654] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:37.807857 5518 tablet_bootstrap.cc:492] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d: No bootstrap required, opened a new log
I20260925 10:32:37.807978 5519 tablet_bootstrap.cc:654] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:37.808156 5518 ts_tablet_manager.cc:1397] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 10:32:37.809204 5519 log.cc:824] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:37.809252 5520 log.cc:824] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:37.812137 5520 tablet_bootstrap.cc:492] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d: No bootstrap required, opened a new log
I20260925 10:32:37.812379 5520 ts_tablet_manager.cc:1397] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 10:32:37.812853 5519 tablet_bootstrap.cc:492] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5: No bootstrap required, opened a new log
I20260925 10:32:37.813103 5519 ts_tablet_manager.cc:1397] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260925 10:32:37.817847 5518 raft_consensus.cc:348] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6683de89f54a10a726e64c7140b40d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39845 } } peers { permanent_uuid: "b212fae77339447a96f16f629c26303d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46749 } } peers { permanent_uuid: "e062531cf4134c1aa298c25ce138f8f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39227 } }
I20260925 10:32:37.818328 5518 raft_consensus.cc:374] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:37.818425 5518 raft_consensus.cc:729] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b212fae77339447a96f16f629c26303d, State: Initialized, Role: FOLLOWER
I20260925 10:32:37.819082 5518 consensus_queue.cc:260] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6683de89f54a10a726e64c7140b40d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39845 } } peers { permanent_uuid: "b212fae77339447a96f16f629c26303d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46749 } } peers { permanent_uuid: "e062531cf4134c1aa298c25ce138f8f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39227 } }
I20260925 10:32:37.820142 5239 heartbeater.cc:503] Master 127.0.97.62:42465 was elected leader, sending a full tablet report...
I20260925 10:32:37.820376 5520 raft_consensus.cc:348] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6683de89f54a10a726e64c7140b40d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39845 } } peers { permanent_uuid: "b212fae77339447a96f16f629c26303d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46749 } } peers { permanent_uuid: "e062531cf4134c1aa298c25ce138f8f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39227 } }
I20260925 10:32:37.820693 5518 ts_tablet_manager.cc:1428] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260925 10:32:37.820749 5520 raft_consensus.cc:374] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:37.820853 5520 raft_consensus.cc:729] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6683de89f54a10a726e64c7140b40d, State: Initialized, Role: FOLLOWER
I20260925 10:32:37.821511 5520 consensus_queue.cc:260] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6683de89f54a10a726e64c7140b40d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39845 } } peers { permanent_uuid: "b212fae77339447a96f16f629c26303d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46749 } } peers { permanent_uuid: "e062531cf4134c1aa298c25ce138f8f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39227 } }
I20260925 10:32:37.822594 5370 heartbeater.cc:503] Master 127.0.97.62:42465 was elected leader, sending a full tablet report...
I20260925 10:32:37.823071 5519 raft_consensus.cc:348] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6683de89f54a10a726e64c7140b40d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39845 } } peers { permanent_uuid: "b212fae77339447a96f16f629c26303d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46749 } } peers { permanent_uuid: "e062531cf4134c1aa298c25ce138f8f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39227 } }
I20260925 10:32:37.823515 5519 raft_consensus.cc:374] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:37.823637 5519 raft_consensus.cc:729] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e062531cf4134c1aa298c25ce138f8f5, State: Initialized, Role: FOLLOWER
I20260925 10:32:37.824190 5519 consensus_queue.cc:260] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6683de89f54a10a726e64c7140b40d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39845 } } peers { permanent_uuid: "b212fae77339447a96f16f629c26303d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46749 } } peers { permanent_uuid: "e062531cf4134c1aa298c25ce138f8f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39227 } }
I20260925 10:32:37.824333 5520 ts_tablet_manager.cc:1428] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260925 10:32:37.825181 5500 heartbeater.cc:503] Master 127.0.97.62:42465 was elected leader, sending a full tablet report...
I20260925 10:32:37.826930 5519 ts_tablet_manager.cc:1428] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260925 10:32:37.877568 5524 raft_consensus.cc:482] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:32:37.877810 5524 raft_consensus.cc:504] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6683de89f54a10a726e64c7140b40d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39845 } } peers { permanent_uuid: "b212fae77339447a96f16f629c26303d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46749 } } peers { permanent_uuid: "e062531cf4134c1aa298c25ce138f8f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39227 } }
I20260925 10:32:37.879526 5524 leader_election.cc:288] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c6683de89f54a10a726e64c7140b40d (127.0.97.2:39845), e062531cf4134c1aa298c25ce138f8f5 (127.0.97.3:39227)
I20260925 10:32:37.882719 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.879743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57915 (local address 127.0.97.2:39845)
0925 10:32:37.879903 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:37.879909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:37.880658 (+ 749us) server_negotiation.cc:407] Connection header received
0925 10:32:37.880838 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:37.880845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:37.880913 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:37.881014 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:37.882121 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:37.882126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:37.882131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:37.882149 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:37.882182 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:37.882196 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:37.882340 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:37.882431 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 10:32:37.882479 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:32:37.882719 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.879922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32771 (local address 127.0.97.3:39227)
0925 10:32:37.880087 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:37.880095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:37.880789 (+ 694us) server_negotiation.cc:407] Connection header received
0925 10:32:37.880906 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:37.880910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:37.880962 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:37.881047 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:37.882121 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:37.882126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:37.882131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:37.882149 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:37.882182 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:37.882196 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:37.882341 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:37.882417 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 10:32:37.882469 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:32:37.883041 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.879862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39227 (local address 127.0.97.1:32771)
0925 10:32:37.880675 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:37.880690 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:37.880783 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:37.881072 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:37.881078 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:37.881102 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:37.881892 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:32:37.881897 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:37.881916 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:32:37.881962 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:37.882293 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:37.882298 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:37.882303 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:32:37.882489 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 10:32:37.882564 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":21,"thread_start_us":417,"threads_started":1}
I20260925 10:32:37.883109 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.879661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39845 (local address 127.0.97.1:57915)
0925 10:32:37.880542 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:37.880561 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:37.880690 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:37.881072 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:37.881078 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:37.881102 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:37.881924 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:32:37.881927 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:37.881938 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:32:37.881962 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:37.882293 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:37.882298 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:37.882303 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:32:37.882496 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 10:32:37.882552 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":444,"threads_started":1}
I20260925 10:32:37.883929 5455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbe763ba3bf4fbfa5846b2115d2a7b0" candidate_uuid: "b212fae77339447a96f16f629c26303d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e062531cf4134c1aa298c25ce138f8f5" is_pre_election: true
I20260925 10:32:37.883971 5325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbe763ba3bf4fbfa5846b2115d2a7b0" candidate_uuid: "b212fae77339447a96f16f629c26303d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6683de89f54a10a726e64c7140b40d" is_pre_election: true
I20260925 10:32:37.884359 5455 raft_consensus.cc:2454] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b212fae77339447a96f16f629c26303d in term 0.
I20260925 10:32:37.884377 5325 raft_consensus.cc:2454] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b212fae77339447a96f16f629c26303d in term 0.
I20260925 10:32:37.884918 5129 leader_election.cc:302] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b212fae77339447a96f16f629c26303d, e062531cf4134c1aa298c25ce138f8f5; no voters:
I20260925 10:32:37.885245 5524 raft_consensus.cc:2781] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:32:37.885371 5524 raft_consensus.cc:482] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:32:37.885499 5524 raft_consensus.cc:3037] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:37.887792 5524 raft_consensus.cc:504] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6683de89f54a10a726e64c7140b40d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39845 } } peers { permanent_uuid: "b212fae77339447a96f16f629c26303d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46749 } } peers { permanent_uuid: "e062531cf4134c1aa298c25ce138f8f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39227 } }
I20260925 10:32:37.888346 5524 leader_election.cc:288] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [CANDIDATE]: Term 1 election: Requested vote from peers 4c6683de89f54a10a726e64c7140b40d (127.0.97.2:39845), e062531cf4134c1aa298c25ce138f8f5 (127.0.97.3:39227)
I20260925 10:32:37.888788 5455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbe763ba3bf4fbfa5846b2115d2a7b0" candidate_uuid: "b212fae77339447a96f16f629c26303d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e062531cf4134c1aa298c25ce138f8f5"
I20260925 10:32:37.888787 5325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbe763ba3bf4fbfa5846b2115d2a7b0" candidate_uuid: "b212fae77339447a96f16f629c26303d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6683de89f54a10a726e64c7140b40d"
I20260925 10:32:37.888962 5455 raft_consensus.cc:3037] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:37.888980 5325 raft_consensus.cc:3037] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:37.891121 5455 raft_consensus.cc:2454] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b212fae77339447a96f16f629c26303d in term 1.
I20260925 10:32:37.891129 5325 raft_consensus.cc:2454] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b212fae77339447a96f16f629c26303d in term 1.
I20260925 10:32:37.891477 5129 leader_election.cc:302] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b212fae77339447a96f16f629c26303d, e062531cf4134c1aa298c25ce138f8f5; no voters:
I20260925 10:32:37.891737 5524 raft_consensus.cc:2781] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:37.892132 5524 raft_consensus.cc:686] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [term 1 LEADER]: Becoming Leader. State: Replica: b212fae77339447a96f16f629c26303d, State: Running, Role: LEADER
I20260925 10:32:37.892546 5524 consensus_queue.cc:237] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6683de89f54a10a726e64c7140b40d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39845 } } peers { permanent_uuid: "b212fae77339447a96f16f629c26303d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46749 } } peers { permanent_uuid: "e062531cf4134c1aa298c25ce138f8f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39227 } }
I20260925 10:32:37.897022 5052 catalog_manager.cc:5495] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d reported cstate change: term changed from 0 to 1, leader changed from <none> to b212fae77339447a96f16f629c26303d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b212fae77339447a96f16f629c26303d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6683de89f54a10a726e64c7140b40d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b212fae77339447a96f16f629c26303d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e062531cf4134c1aa298c25ce138f8f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39227 } health_report { overall_health: UNKNOWN } } }
W20260925 10:32:37.906680 5240 tablet.cc:2369] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:32:37.936826 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.935063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41392 (local address 127.0.97.1:46749)
0925 10:32:37.935251 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:37.935255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:37.935274 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:32:37.935446 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:37.935449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:37.935515 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:37.935600 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:37.936304 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:37.936309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:37.936311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:37.936324 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:37.936350 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:37.936361 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:37.936453 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:37.936515 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 10:32:37.936554 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:32:37.944041 5455 raft_consensus.cc:1261] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5 [term 1 FOLLOWER]: Refusing update from remote peer b212fae77339447a96f16f629c26303d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:37.944078 5325 raft_consensus.cc:1261] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d [term 1 FOLLOWER]: Refusing update from remote peer b212fae77339447a96f16f629c26303d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:37.944901 5529 consensus_queue.cc:1035] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c6683de89f54a10a726e64c7140b40d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:37.945215 5524 consensus_queue.cc:1035] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e062531cf4134c1aa298c25ce138f8f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:37.960897 5533 mvcc.cc:205] Tried to move back new op lower bound from 7333201338129956864 to 7333201337931350016. Current Snapshot: MvccSnapshot[applied={T|T < 7333201338129956864}]
I20260925 10:32:37.976071 5052 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:37.976362 5052 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:37.980238 5052 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 203955ffa5ff4cc49873203ae0832dea: Sending DeleteTablet for 3 replicas of tablet dbbe763ba3bf4fbfa5846b2115d2a7b0
I20260925 10:32:37.981256 5305 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbe763ba3bf4fbfa5846b2115d2a7b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:37 UTC) from {username='slave'} at 127.0.0.1:43602
I20260925 10:32:37.981426 5174 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbe763ba3bf4fbfa5846b2115d2a7b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:37 UTC) from {username='slave'} at 127.0.0.1:41390
I20260925 10:32:37.982110 5542 tablet_replica.cc:330] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d: stopping tablet replica
I20260925 10:32:37.982685 5542 raft_consensus.cc:2229] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:37.983294 5542 raft_consensus.cc:2258] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:37.984593 5173 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbe763ba3bf4fbfa5846b2115d2a7b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:37 UTC) from {username='slave'} at 127.0.0.1:41390
I20260925 10:32:37.985473 5435 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbe763ba3bf4fbfa5846b2115d2a7b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:37 UTC) from {username='slave'} at 127.0.0.1:58996
I20260925 10:32:37.985651 5542 ts_tablet_manager.cc:1905] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:37.988195 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 5112
I20260925 10:32:37.994470 5542 ts_tablet_manager.cc:1918] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:32:37.994720 5542 log.cc:1196] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/dbbe763ba3bf4fbfa5846b2115d2a7b0
I20260925 10:32:37.995344 5542 ts_tablet_manager.cc:1939] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P 4c6683de89f54a10a726e64c7140b40d: Deleting consensus metadata
I20260925 10:32:37.997267 5037 catalog_manager.cc:4841] TS 4c6683de89f54a10a726e64c7140b40d (127.0.97.2:39845): tablet dbbe763ba3bf4fbfa5846b2115d2a7b0 (table test-table [id=a2bb0527fed848779863d1ceb942b309]) successfully deleted
I20260925 10:32:37.999514 5543 tablet_replica.cc:330] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d: stopping tablet replica
I20260925 10:32:37.999945 5543 raft_consensus.cc:2229] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:32:38.000685 5543 raft_consensus.cc:2258] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:38.001108 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:37.990235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.0.97.1:46749)
0925 10:32:37.990410 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:37.990416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:37.998828 (+ 8412us) server_negotiation.cc:407] Connection header received
0925 10:32:37.999076 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:37.999081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:37.999144 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:37.999243 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:38.000313 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:38.000318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:38.000324 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:38.000343 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:38.000381 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:38.000395 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:38.000516 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:38.000771 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 10:32:38.000830 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:32:38.002452 5234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:38.002125 5544 tablet_replica.cc:330] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5: stopping tablet replica
I20260925 10:32:38.003280 5544 raft_consensus.cc:2229] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:38.003846 5544 raft_consensus.cc:2258] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:38.005801 5544 ts_tablet_manager.cc:1905] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:38.011358 5544 ts_tablet_manager.cc:1918] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:32:38.011524 5544 log.cc:1196] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/dbbe763ba3bf4fbfa5846b2115d2a7b0
I20260925 10:32:38.011974 5544 ts_tablet_manager.cc:1939] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P e062531cf4134c1aa298c25ce138f8f5: Deleting consensus metadata
I20260925 10:32:38.013353 5039 catalog_manager.cc:4841] TS e062531cf4134c1aa298c25ce138f8f5 (127.0.97.3:39227): tablet dbbe763ba3bf4fbfa5846b2115d2a7b0 (table test-table [id=a2bb0527fed848779863d1ceb942b309]) successfully deleted
I20260925 10:32:38.002995 5543 ts_tablet_manager.cc:1905] T dbbe763ba3bf4fbfa5846b2115d2a7b0 P b212fae77339447a96f16f629c26303d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:38.094951 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 5112
W20260925 10:32:38.111688 5038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:46749 (error 108)
I20260925 10:32:38.111855 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 5243
I20260925 10:32:38.113623 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:38.113190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46749 (local address 127.0.0.1:41400)
0925 10:32:38.113352 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:38.113492 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 10:32:38.113849 5038 catalog_manager.cc:4584] TS b212fae77339447a96f16f629c26303d (127.0.97.1:46749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbbe763ba3bf4fbfa5846b2115d2a7b0: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46749: connect: Connection refused (error 111)
I20260925 10:32:38.114717 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:38.112733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.0.97.2:39845)
0925 10:32:38.112929 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:38.112935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:38.112957 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:32:38.113093 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:38.113099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:38.113185 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:38.113307 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:38.114223 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:38.114227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:38.114230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:38.114243 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:38.114270 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:38.114281 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:38.114437 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:38.114492 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 10:32:38.114533 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:32:38.115325 5365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:38.158295 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:38.157853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46749 (local address 127.0.0.1:41406)
0925 10:32:38.158060 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:38.158150 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 10:32:38.159235 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:38.158932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46749 (local address 127.0.0.1:41414)
0925 10:32:38.159064 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:38.159131 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 10:32:38.167179 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:38.166846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46749 (local address 127.0.0.1:41430)
0925 10:32:38.166994 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:38.167073 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 10:32:38.167977 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:38.167699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46749 (local address 127.0.0.1:41436)
0925 10:32:38.167822 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:38.167889 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 10:32:38.201306 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 5243
I20260925 10:32:38.208257 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:38.207788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46749 (local address 127.0.0.1:41440)
0925 10:32:38.208003 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:38.208098 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 10:32:38.209280 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:38.208859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46749 (local address 127.0.0.1:41456)
0925 10:32:38.209115 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:38.209175 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260925 10:32:38.217685 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 5373
I20260925 10:32:38.220460 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:38.218414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.0.97.3:39227)
0925 10:32:38.218595 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:38.218601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:38.218619 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:32:38.218810 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:38.218821 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:38.218890 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:38.219000 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:38.219842 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:38.219847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:38.219852 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:38.219868 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:38.219906 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:38.219917 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:38.220052 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:38.220126 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 10:32:38.220182 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:32:38.221040 5495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:38.244547 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:38.244069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46749 (local address 127.0.0.1:41468)
0925 10:32:38.244273 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:38.244368 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 10:32:38.245523 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:38.245146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46749 (local address 127.0.0.1:41474)
0925 10:32:38.245302 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:38.245384 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 10:32:38.305989 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 5373
I20260925 10:32:38.309894 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:38.309482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46749 (local address 127.0.0.1:41478)
0925 10:32:38.309656 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:38.309738 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 10:32:38.310791 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:38.310491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46749 (local address 127.0.0.1:41486)
0925 10:32:38.310623 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:38.310693 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 10:32:38.322499 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 5022
I20260925 10:32:38.323444 5083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:38.426226 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:38.425546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46749 (local address 127.0.0.1:41500)
0925 10:32:38.425928 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:38.426023 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214}
I20260925 10:32:38.426524 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 5022
2026-09-25T10:32:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2498 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 10:32:38.462451 388 mini_oidc.cc:152] Starting JWKS server
I20260925 10:32:38.464799 388 webserver.cc:460] Webserver started at https://127.0.0.1:33595/ using document root <none> and password file <none>
I20260925 10:32:38.465042 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:32:38.465565 388 webserver.cc:460] Webserver started at http://127.0.0.1:42289/ using document root <none> and password file <none>
2026-09-25T10:32:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:32:38Z Disabled control of system clock
I20260925 10:32:38.489494 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42919
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42919
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33595/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/testchainca.cert with env {}
W20260925 10:32:38.795146 5556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:38.795480 5556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:38.795656 5556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:38.802553 5556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:38.802685 5556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:38.802766 5556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:38.802870 5556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:38.802953 5556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:32:38.803040 5556 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:32:38.803114 5556 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33595/jwks.json
I20260925 10:32:38.812505 5556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42919
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33595/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:38.813764 5556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:38.815007 5556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:38.826646 5561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:38.826579 5562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:38.826684 5564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:38.827435 5556 server_base.cc:1034] running on GCE node
I20260925 10:32:38.828154 5556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:38.829397 5556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:38.830622 5556 hybrid_clock.cc:648] HybridClock initialized: now 1790332358830570 us; error 58 us; skew 500 ppm
I20260925 10:32:38.830951 5556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:32:38.832991 5556 webserver.cc:460] Webserver started at http://127.0.97.62:46249/ using document root <none> and password file <none>
I20260925 10:32:38.833618 5556 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:38.833745 5556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:38.834061 5556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:38.836755 5556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "998aa32815484b41907b61cfaaca7c68"
format_stamp: "Formatted at 2026-09-25 10:32:38 on dist-test-slave-dxdd"
server_key: "ebe1d41ad5eeefc31126a328472b2ebf"
server_key_iv: "05be07ec20ef68f5bccc981d0c01cc94"
server_key_version: "encryptionkey@0"
I20260925 10:32:38.837554 5556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "998aa32815484b41907b61cfaaca7c68"
format_stamp: "Formatted at 2026-09-25 10:32:38 on dist-test-slave-dxdd"
server_key: "ebe1d41ad5eeefc31126a328472b2ebf"
server_key_iv: "05be07ec20ef68f5bccc981d0c01cc94"
server_key_version: "encryptionkey@0"
I20260925 10:32:38.842188 5556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 10:32:38.845322 5570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:38.846717 5556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:32:38.846892 5556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "998aa32815484b41907b61cfaaca7c68"
format_stamp: "Formatted at 2026-09-25 10:32:38 on dist-test-slave-dxdd"
server_key: "ebe1d41ad5eeefc31126a328472b2ebf"
server_key_iv: "05be07ec20ef68f5bccc981d0c01cc94"
server_key_version: "encryptionkey@0"
I20260925 10:32:38.847098 5556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:38.867735 5556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:38.868458 5556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:38.868722 5556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:38.885550 5556 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42919
I20260925 10:32:38.885568 5621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42919 every 8 connection(s)
I20260925 10:32:38.886831 5556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:32:38.890280 5622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:38.894294 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 5556
I20260925 10:32:38.894555 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:32:38.895124 388 external_mini_cluster.cc:1428] Setting key c1cbfe30ffc4c5e93b0c89026d010495
I20260925 10:32:38.900316 5622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68: Bootstrap starting.
I20260925 10:32:38.903546 5622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:38.904613 5622 log.cc:824] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:38.907423 5622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68: No bootstrap required, opened a new log
I20260925 10:32:38.907893 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:38.896730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32964 (local address 127.0.97.62:42919)
0925 10:32:38.897613 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:38.897623 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:38.897663 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:32:38.899320 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:38.899341 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:38.899726 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:38.900031 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:38.900618 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:38.901849 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:38.903744 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:38.904359 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:38.904488 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:38.905725 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:38.905735 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:38.905755 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:38.906283 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:38.906403 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:38.906422 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:38.906506 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:38.906709 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 10:32:38.907021 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":350,"threads_started":1}
I20260925 10:32:38.911540 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:32:38.916661 5622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998aa32815484b41907b61cfaaca7c68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42919 } }
I20260925 10:32:38.916941 5622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:38.917008 5622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998aa32815484b41907b61cfaaca7c68, State: Initialized, Role: FOLLOWER
I20260925 10:32:38.917443 5622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998aa32815484b41907b61cfaaca7c68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42919 } }
I20260925 10:32:38.917629 5622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:32:38.917716 5622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:32:38.917829 5622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:38.919821 5622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998aa32815484b41907b61cfaaca7c68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42919 } }
I20260925 10:32:38.920325 5622 leader_election.cc:302] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 998aa32815484b41907b61cfaaca7c68; no voters:
I20260925 10:32:38.920892 5622 leader_election.cc:288] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:32:38.921058 5627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:38.921691 5627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [term 1 LEADER]: Becoming Leader. State: Replica: 998aa32815484b41907b61cfaaca7c68, State: Running, Role: LEADER
I20260925 10:32:38.922187 5627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998aa32815484b41907b61cfaaca7c68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42919 } }
I20260925 10:32:38.922945 5622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:38.925136 5628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "998aa32815484b41907b61cfaaca7c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998aa32815484b41907b61cfaaca7c68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42919 } } }
I20260925 10:32:38.925176 5629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 998aa32815484b41907b61cfaaca7c68. Latest consensus state: current_term: 1 leader_uuid: "998aa32815484b41907b61cfaaca7c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998aa32815484b41907b61cfaaca7c68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42919 } } }
I20260925 10:32:38.925487 5628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:38.925592 5629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:38.932147 5632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:38.937094 5632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:38.945423 5632 catalog_manager.cc:1347] Generated new cluster ID: 8b443254830447ac92d7a890d63ac9a1
I20260925 10:32:38.945554 5632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:38.957444 5632 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:32:38.958707 5632 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:38.965495 5632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 998aa32815484b41907b61cfaaca7c68: Generated new TSK 0
I20260925 10:32:38.966238 5632 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 10:32:39.006095 5652 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 10:32:39.006249 5652 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 10:32:39.007418 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:38.986796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.0.97.62:42919)
0925 10:32:38.986989 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:38.986995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:38.987425 (+ 430us) server_negotiation.cc:407] Connection header received
0925 10:32:38.987681 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:38.987688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:38.987768 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:38.987919 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:32:38.988363 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:38.989458 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:38.990898 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:38.991599 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:38.991716 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:38.992187 (+ 471us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:32:39.006769 (+ 14582us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 10:32:39.007117 (+ 348us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32980: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 10:32:39.007524 5625 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32980: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 10:32:39.008633 5651 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:32:38.986693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42919 (local address 127.0.0.1:32980)
0925 10:32:38.987329 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:38.987346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:38.987492 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:38.987948 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:38.987952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:38.987960 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:32:38.988231 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:38.988237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:38.989577 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:38.989581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:38.990759 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:38.990771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:38.991723 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:38.991759 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:38.991882 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:38.991929 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:32:39.007023 (+ 15094us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 10:32:39.007240 (+ 217us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42919: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":352,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:42919: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 10:32:39.010571 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 5556
I20260925 10:32:39.011601 5617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:39.102134 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 5556
2026-09-25T10:32:39Z chronyd exiting
I20260925 10:32:39.148396 388 test_util.cc:175] -----------------------------------------------
I20260925 10:32:39.148556 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332315635355-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (690 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 10:32:39.152240 388 mini_oidc.cc:152] Starting JWKS server
I20260925 10:32:39.153909 388 webserver.cc:460] Webserver started at https://127.0.0.1:37359/ using document root <none> and password file <none>
I20260925 10:32:39.154141 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:32:39.154812 388 webserver.cc:460] Webserver started at http://127.0.0.1:33155/ using document root <none> and password file <none>
2026-09-25T10:32:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:32:39Z Disabled control of system clock
I20260925 10:32:39.180020 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45941
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45941
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37359/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/testchainca.cert with env {}
W20260925 10:32:39.486125 5659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:39.486449 5659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:39.486552 5659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:39.493292 5659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:39.493377 5659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:39.493413 5659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:39.493464 5659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:39.493506 5659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:32:39.493585 5659 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:32:39.493645 5659 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37359/jwks.json
I20260925 10:32:39.502871 5659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45941
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37359/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:39.503978 5659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:39.505110 5659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:39.516178 5665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:39.516116 5664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:39.516127 5667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:39.516764 5659 server_base.cc:1034] running on GCE node
I20260925 10:32:39.517462 5659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:39.518756 5659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:39.519946 5659 hybrid_clock.cc:648] HybridClock initialized: now 1790332359519910 us; error 39 us; skew 500 ppm
I20260925 10:32:39.520277 5659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:32:39.522286 5659 webserver.cc:460] Webserver started at http://127.0.97.62:33609/ using document root <none> and password file <none>
I20260925 10:32:39.522960 5659 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:39.523070 5659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:39.523386 5659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:39.525981 5659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "e4e0506175ce4113a1b7551dbdc1ae04"
format_stamp: "Formatted at 2026-09-25 10:32:39 on dist-test-slave-dxdd"
server_key: "64998e6f74e8b6fc92d17807f76a9bc9"
server_key_iv: "60884789f5e5623c7987100e99b40dd5"
server_key_version: "encryptionkey@0"
I20260925 10:32:39.526805 5659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "e4e0506175ce4113a1b7551dbdc1ae04"
format_stamp: "Formatted at 2026-09-25 10:32:39 on dist-test-slave-dxdd"
server_key: "64998e6f74e8b6fc92d17807f76a9bc9"
server_key_iv: "60884789f5e5623c7987100e99b40dd5"
server_key_version: "encryptionkey@0"
I20260925 10:32:39.531332 5659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:32:39.534457 5673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:39.535625 5659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 10:32:39.535804 5659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "e4e0506175ce4113a1b7551dbdc1ae04"
format_stamp: "Formatted at 2026-09-25 10:32:39 on dist-test-slave-dxdd"
server_key: "64998e6f74e8b6fc92d17807f76a9bc9"
server_key_iv: "60884789f5e5623c7987100e99b40dd5"
server_key_version: "encryptionkey@0"
I20260925 10:32:39.535998 5659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:39.553010 5659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:39.553709 5659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:39.553990 5659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:39.571231 5659 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45941
I20260925 10:32:39.571264 5724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45941 every 8 connection(s)
I20260925 10:32:39.572404 5659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:32:39.574954 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 5659
I20260925 10:32:39.575137 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:32:39.575758 388 external_mini_cluster.cc:1428] Setting key 4eb3a4455ec29cd6b8fb522ddd40b1e3
I20260925 10:32:39.575889 5725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:39.588373 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:39.577390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39878 (local address 127.0.97.62:45941)
0925 10:32:39.578294 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:39.578353 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:39.578401 (+ 48us) server_negotiation.cc:407] Connection header received
0925 10:32:39.579779 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:39.579797 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:39.580237 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:39.580570 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:39.581111 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:39.582519 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:39.584343 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:39.584946 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:39.585068 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:39.586242 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:39.586249 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:39.586261 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:39.586842 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:39.586911 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:39.586929 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:39.587023 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:39.587240 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 10:32:39.587555 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":361,"threads_started":1}
I20260925 10:32:39.589694 5725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04: Bootstrap starting.
I20260925 10:32:39.591892 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:32:39.592823 5725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:39.593746 5725 log.cc:824] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:39.596408 5725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04: No bootstrap required, opened a new log
I20260925 10:32:39.603088 5725 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e0506175ce4113a1b7551dbdc1ae04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45941 } }
I20260925 10:32:39.603440 5725 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:39.603546 5725 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4e0506175ce4113a1b7551dbdc1ae04, State: Initialized, Role: FOLLOWER
I20260925 10:32:39.604058 5725 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e0506175ce4113a1b7551dbdc1ae04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45941 } }
I20260925 10:32:39.604292 5725 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:32:39.604427 5725 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:32:39.604596 5725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:39.606621 5725 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e0506175ce4113a1b7551dbdc1ae04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45941 } }
I20260925 10:32:39.607069 5725 leader_election.cc:302] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4e0506175ce4113a1b7551dbdc1ae04; no voters:
I20260925 10:32:39.607709 5725 leader_election.cc:288] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:32:39.607863 5730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:39.608647 5730 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [term 1 LEADER]: Becoming Leader. State: Replica: e4e0506175ce4113a1b7551dbdc1ae04, State: Running, Role: LEADER
I20260925 10:32:39.609215 5730 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e0506175ce4113a1b7551dbdc1ae04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45941 } }
I20260925 10:32:39.609831 5725 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:39.612351 5732 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4e0506175ce4113a1b7551dbdc1ae04. Latest consensus state: current_term: 1 leader_uuid: "e4e0506175ce4113a1b7551dbdc1ae04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e0506175ce4113a1b7551dbdc1ae04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45941 } } }
I20260925 10:32:39.612334 5731 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4e0506175ce4113a1b7551dbdc1ae04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e0506175ce4113a1b7551dbdc1ae04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45941 } } }
I20260925 10:32:39.612833 5732 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:39.612851 5731 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:39.614410 5738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:39.619341 5738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:39.627826 5738 catalog_manager.cc:1347] Generated new cluster ID: f8e1a03724704fdead945f930bb04fe8
I20260925 10:32:39.627938 5738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:39.649257 5738 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:32:39.650540 5738 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:39.668431 5738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4e0506175ce4113a1b7551dbdc1ae04: Generated new TSK 0
I20260925 10:32:39.669180 5738 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 10:32:39.698160 5755 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 10:32:39.698271 5755 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 10:32:39.699402 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:39.680319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.0.97.62:45941)
0925 10:32:39.680490 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:39.680496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:39.680933 (+ 437us) server_negotiation.cc:407] Connection header received
0925 10:32:39.681146 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:39.681151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:39.681247 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:39.681393 (+ 146us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:32:39.681787 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:39.682839 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:39.684157 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:39.684814 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:39.684939 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:39.685451 (+ 512us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:32:39.698783 (+ 13332us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37359/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 10:32:39.699093 (+ 310us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39890: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37359/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 10:32:39.699702 5754 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:32:39.680140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45941 (local address 127.0.0.1:39890)
0925 10:32:39.680821 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:39.680836 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:39.680970 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:39.681405 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:39.681408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:39.681417 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:32:39.681660 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:39.681666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:39.682971 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:39.682974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:39.684029 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:39.684039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:39.684935 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:39.684997 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:39.685161 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:39.685203 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:32:39.698989 (+ 13786us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37359/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 10:32:39.699246 (+ 257us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45941: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37359/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":402,"threads_started":1}
W20260925 10:32:39.699740 5728 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39890: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37359/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260925 10:32:39.700827 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 5659
I20260925 10:32:39.702093 5720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:39.793041 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 5659
2026-09-25T10:32:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (697 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 10:32:39.849390 388 mini_oidc.cc:152] Starting JWKS server
I20260925 10:32:39.851234 388 webserver.cc:460] Webserver started at https://127.0.0.1:37755/ using document root <none> and password file <none>
I20260925 10:32:39.851486 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:32:39.852057 388 webserver.cc:460] Webserver started at http://127.0.0.1:45651/ using document root <none> and password file <none>
2026-09-25T10:32:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:32:39Z Disabled control of system clock
I20260925 10:32:39.876621 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33415
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33415
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37755/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 10:32:40.178772 5762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:40.179126 5762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:40.179276 5762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:40.186352 5762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:40.186486 5762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:40.186569 5762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:40.186676 5762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:40.186748 5762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:32:40.186830 5762 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:32:40.186924 5762 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37755/jwks.json
I20260925 10:32:40.195940 5762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33415
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37755/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:40.197229 5762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:40.198990 5762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:40.210901 5767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:40.210937 5770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:40.210966 5768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:40.212018 5762 server_base.cc:1034] running on GCE node
I20260925 10:32:40.212745 5762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:40.214013 5762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:40.215219 5762 hybrid_clock.cc:648] HybridClock initialized: now 1790332360215179 us; error 40 us; skew 500 ppm
I20260925 10:32:40.215566 5762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:32:40.224428 5762 webserver.cc:460] Webserver started at http://127.0.97.62:42331/ using document root <none> and password file <none>
I20260925 10:32:40.225106 5762 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:40.225227 5762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:40.225570 5762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:40.228404 5762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "49d9d0d85405451fb6df52bea158e55d"
format_stamp: "Formatted at 2026-09-25 10:32:40 on dist-test-slave-dxdd"
server_key: "fb70d5a2e8d5b8b1a8875cba8a48937a"
server_key_iv: "26546f7b56ac92be167e6f92fe9b4e75"
server_key_version: "encryptionkey@0"
I20260925 10:32:40.229166 5762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "49d9d0d85405451fb6df52bea158e55d"
format_stamp: "Formatted at 2026-09-25 10:32:40 on dist-test-slave-dxdd"
server_key: "fb70d5a2e8d5b8b1a8875cba8a48937a"
server_key_iv: "26546f7b56ac92be167e6f92fe9b4e75"
server_key_version: "encryptionkey@0"
I20260925 10:32:40.233949 5762 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 10:32:40.237411 5776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:40.238752 5762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:32:40.238922 5762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "49d9d0d85405451fb6df52bea158e55d"
format_stamp: "Formatted at 2026-09-25 10:32:40 on dist-test-slave-dxdd"
server_key: "fb70d5a2e8d5b8b1a8875cba8a48937a"
server_key_iv: "26546f7b56ac92be167e6f92fe9b4e75"
server_key_version: "encryptionkey@0"
I20260925 10:32:40.239122 5762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:40.270284 5762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:40.271121 5762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:40.271409 5762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:40.288606 5762 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33415
I20260925 10:32:40.288641 5827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33415 every 8 connection(s)
I20260925 10:32:40.289903 5762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:32:40.293514 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 5762
I20260925 10:32:40.293572 5828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:40.293746 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:32:40.294621 388 external_mini_cluster.cc:1428] Setting key d15aff88c2ff929b82ad7690a062b950
I20260925 10:32:40.306715 5828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d: Bootstrap starting.
I20260925 10:32:40.308105 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:40.296413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.0.97.62:33415)
0925 10:32:40.297428 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:40.297441 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:40.297481 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:32:40.299555 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:40.299576 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:40.300017 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:40.300339 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:40.300854 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:40.302038 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:40.303892 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:40.304593 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:40.304725 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:40.305878 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:40.305885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:40.305898 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:40.306460 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:40.306529 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:40.306552 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:40.306658 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:40.306962 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 10:32:40.307262 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":398,"threads_started":1}
I20260925 10:32:40.309706 5828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:40.310928 5828 log.cc:824] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:40.311812 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:32:40.313509 5828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d: No bootstrap required, opened a new log
I20260925 10:32:40.320374 5828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d9d0d85405451fb6df52bea158e55d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } }
I20260925 10:32:40.320701 5828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:40.320827 5828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d9d0d85405451fb6df52bea158e55d, State: Initialized, Role: FOLLOWER
I20260925 10:32:40.321341 5828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d9d0d85405451fb6df52bea158e55d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } }
I20260925 10:32:40.321566 5828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:32:40.321704 5828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:32:40.321866 5828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:40.323935 5828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d9d0d85405451fb6df52bea158e55d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } }
I20260925 10:32:40.324399 5828 leader_election.cc:302] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49d9d0d85405451fb6df52bea158e55d; no voters:
I20260925 10:32:40.324911 5828 leader_election.cc:288] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:32:40.325078 5833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:40.325790 5833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [term 1 LEADER]: Becoming Leader. State: Replica: 49d9d0d85405451fb6df52bea158e55d, State: Running, Role: LEADER
I20260925 10:32:40.326277 5833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d9d0d85405451fb6df52bea158e55d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } }
I20260925 10:32:40.326984 5828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:40.329075 5835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49d9d0d85405451fb6df52bea158e55d. Latest consensus state: current_term: 1 leader_uuid: "49d9d0d85405451fb6df52bea158e55d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d9d0d85405451fb6df52bea158e55d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } } }
I20260925 10:32:40.329532 5835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:40.330150 5834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49d9d0d85405451fb6df52bea158e55d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d9d0d85405451fb6df52bea158e55d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } } }
I20260925 10:32:40.330592 5834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:40.331923 5842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:40.336797 5842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:40.348476 5842 catalog_manager.cc:1347] Generated new cluster ID: d9ff20e683d14c539d1d718a02554d09
I20260925 10:32:40.348587 5842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:40.381830 5842 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:32:40.383694 5842 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:40.395750 5842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49d9d0d85405451fb6df52bea158e55d: Generated new TSK 0
I20260925 10:32:40.396528 5842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:32:40.420077 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:40.411898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56852 (local address 127.0.97.62:33415)
0925 10:32:40.412074 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:40.412081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:40.412551 (+ 470us) server_negotiation.cc:407] Connection header received
0925 10:32:40.412758 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:40.412764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:40.412854 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:40.413010 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:32:40.413446 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:40.414627 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:40.415935 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:40.416620 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:40.416743 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:40.417193 (+ 450us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:32:40.419495 (+ 2302us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37755/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 10:32:40.419778 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56852: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37755/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 10:32:40.420194 5831 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56852: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37755/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 10:32:40.421800 5857 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:32:40.411817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33415 (local address 127.0.0.1:56852)
0925 10:32:40.412464 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:40.412478 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:40.412589 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:40.413020 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:40.413023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:40.413032 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:32:40.413306 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:40.413313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:40.414754 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:40.414758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:40.415793 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:40.415803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:40.416740 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:40.416797 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:40.416919 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:40.416947 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:32:40.419681 (+ 2734us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37755/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 10:32:40.421399 (+ 1718us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:33415: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37755/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":1457,"thread_start_us":380,"threads_started":1}
I20260925 10:32:40.422777 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 5762
I20260925 10:32:40.423892 5823 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 10:32:40.431759 5858 webserver.cc:533] Webserver: sslize failed:
E20260925 10:32:40.431834 5858 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 10:32:40.513758 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 5762
2026-09-25T10:32:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (718 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T10:32:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:32:40Z Disabled control of system clock
I20260925 10:32:40.592499 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43955
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43955
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/testchainca.cert with env {}
W20260925 10:32:40.903553 5863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:40.903820 5863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:40.903905 5863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:40.910784 5863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:40.910869 5863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:40.910907 5863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:40.910961 5863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:40.911003 5863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:32:40.911055 5863 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:32:40.911087 5863 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 10:32:40.920106 5863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:40.921134 5863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:40.922264 5863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:40.933460 5868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:40.933395 5869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:40.933485 5871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:40.934295 5863 server_base.cc:1034] running on GCE node
I20260925 10:32:40.935012 5863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:40.936245 5863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:40.937466 5863 hybrid_clock.cc:648] HybridClock initialized: now 1790332360937438 us; error 50 us; skew 500 ppm
I20260925 10:32:40.937790 5863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:32:40.939947 5863 webserver.cc:460] Webserver started at http://127.0.97.62:40301/ using document root <none> and password file <none>
I20260925 10:32:40.940644 5863 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:40.940779 5863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:40.941087 5863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:40.943956 5863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "09a11c9aa1ee4f17b401fa49d9eff418"
format_stamp: "Formatted at 2026-09-25 10:32:40 on dist-test-slave-dxdd"
server_key: "7a8384d61bf835715df9e01d3f7e4580"
server_key_iv: "e31ff5eb2bb24c5752ab21ca519a7933"
server_key_version: "encryptionkey@0"
I20260925 10:32:40.944779 5863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "09a11c9aa1ee4f17b401fa49d9eff418"
format_stamp: "Formatted at 2026-09-25 10:32:40 on dist-test-slave-dxdd"
server_key: "7a8384d61bf835715df9e01d3f7e4580"
server_key_iv: "e31ff5eb2bb24c5752ab21ca519a7933"
server_key_version: "encryptionkey@0"
I20260925 10:32:40.949920 5863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:32:40.953456 5877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:40.954933 5863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 10:32:40.955122 5863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "09a11c9aa1ee4f17b401fa49d9eff418"
format_stamp: "Formatted at 2026-09-25 10:32:40 on dist-test-slave-dxdd"
server_key: "7a8384d61bf835715df9e01d3f7e4580"
server_key_iv: "e31ff5eb2bb24c5752ab21ca519a7933"
server_key_version: "encryptionkey@0"
I20260925 10:32:40.955356 5863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:40.978857 5863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:40.979620 5863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:40.979897 5863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:40.997368 5863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43955
I20260925 10:32:40.997408 5928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43955 every 8 connection(s)
I20260925 10:32:40.998594 5863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:32:41.002123 5929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:41.008514 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 5863
I20260925 10:32:41.008730 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:32:41.009569 388 external_mini_cluster.cc:1428] Setting key 50a9aefc31d21f5b77d3ca3715546faa
I20260925 10:32:41.013168 5929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418: Bootstrap starting.
I20260925 10:32:41.016400 5929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:41.017591 5929 log.cc:824] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:41.020766 5929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418: No bootstrap required, opened a new log
I20260925 10:32:41.022954 5932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:41.011423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45948 (local address 127.0.97.62:43955)
0925 10:32:41.012369 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:41.012379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:41.012436 (+ 57us) server_negotiation.cc:407] Connection header received
0925 10:32:41.014062 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:41.014082 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:41.014580 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:41.014886 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:41.015457 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:41.016729 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:41.018540 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:41.019341 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:41.019480 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:41.020703 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:41.020710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:41.020722 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:32:41.021300 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:41.021372 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:32:41.021390 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:41.021482 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:41.021689 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 10:32:41.021998 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":367,"threads_started":1}
I20260925 10:32:41.026386 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:32:41.029439 5929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a11c9aa1ee4f17b401fa49d9eff418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43955 } }
I20260925 10:32:41.029788 5929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:41.029884 5929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a11c9aa1ee4f17b401fa49d9eff418, State: Initialized, Role: FOLLOWER
I20260925 10:32:41.030469 5929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a11c9aa1ee4f17b401fa49d9eff418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43955 } }
I20260925 10:32:41.030692 5929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:32:41.030833 5929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:32:41.030993 5929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:41.032974 5929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a11c9aa1ee4f17b401fa49d9eff418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43955 } }
I20260925 10:32:41.033435 5929 leader_election.cc:302] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09a11c9aa1ee4f17b401fa49d9eff418; no voters:
I20260925 10:32:41.033968 5929 leader_election.cc:288] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:32:41.034148 5934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:41.035246 5934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [term 1 LEADER]: Becoming Leader. State: Replica: 09a11c9aa1ee4f17b401fa49d9eff418, State: Running, Role: LEADER
I20260925 10:32:41.035775 5934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a11c9aa1ee4f17b401fa49d9eff418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43955 } }
I20260925 10:32:41.036105 5929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:41.038679 5936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09a11c9aa1ee4f17b401fa49d9eff418. Latest consensus state: current_term: 1 leader_uuid: "09a11c9aa1ee4f17b401fa49d9eff418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a11c9aa1ee4f17b401fa49d9eff418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43955 } } }
I20260925 10:32:41.039016 5936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:41.039525 5935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09a11c9aa1ee4f17b401fa49d9eff418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a11c9aa1ee4f17b401fa49d9eff418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43955 } } }
I20260925 10:32:41.039955 5935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:41.044154 5943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:41.052822 5943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:41.062682 5943 catalog_manager.cc:1347] Generated new cluster ID: 5327b66a9ff0442b9fca441315f37d43
I20260925 10:32:41.062804 5943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:41.078980 5943 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:32:41.080579 5943 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:41.087823 5943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09a11c9aa1ee4f17b401fa49d9eff418: Generated new TSK 0
I20260925 10:32:41.088531 5943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:32:41.548776 5932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:41.104042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45952 (local address 127.0.97.62:43955)
0925 10:32:41.104229 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:41.104236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:41.104672 (+ 436us) server_negotiation.cc:407] Connection header received
0925 10:32:41.104901 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:41.104907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:41.104999 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:41.105196 (+ 197us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:32:41.105624 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:41.106755 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:41.108062 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:41.108726 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:41.108860 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:41.109311 (+ 451us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:32:41.548237 (+438926us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 10:32:41.548551 (+ 314us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45952: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 10:32:41.548875 5932 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45952: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 10:32:41.549167 5958 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:32:41.103960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43955 (local address 127.0.0.1:45952)
0925 10:32:41.104566 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:41.104582 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:41.104708 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:41.105186 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:41.105197 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:41.105207 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:32:41.105481 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:41.105489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:41.106896 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:41.106899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:41.107929 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:41.107940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:41.108854 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:41.108896 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:41.109021 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:41.109050 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:32:41.548541 (+439491us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 10:32:41.548775 (+ 234us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:43955: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":340,"threads_started":1}
I20260925 10:32:41.550437 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 5863
I20260925 10:32:41.551499 5924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:41.653903 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 5863
2026-09-25T10:32:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1121 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (394 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (408 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:32:42 dist-test-slave-dxdd krb5kdc[5973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:32:42 dist-test-slave-dxdd krb5kdc[5973](info): set up 2 sockets
Sep 25 10:32:42 dist-test-slave-dxdd krb5kdc[5973](info): commencing operation
krb5kdc: starting...
W20260925 10:32:43.534772 388 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:32:43 dist-test-slave-dxdd krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:32:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:32:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:32:43.732072 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41877
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:44.050899 5986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:44.051205 5986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:44.051295 5986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:44.058729 5986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:44.058858 5986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:44.058905 5986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:44.058961 5986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:44.059005 5986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:32:44.069170 5986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:44.070550 5986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:44.071836 5986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:44.083279 5991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:44.083348 5992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:44.083323 5994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:44.084395 5986 server_base.cc:1034] running on GCE node
I20260925 10:32:44.085161 5986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:44.086725 5986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:44.087960 5986 hybrid_clock.cc:648] HybridClock initialized: now 1790332364087908 us; error 66 us; skew 500 ppm
I20260925 10:32:44.088296 5986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:44 dist-test-slave-dxdd krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:44.091308 5986 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:32:44.092796 5986 webserver.cc:460] Webserver started at http://127.0.97.62:42195/ using document root <none> and password file <none>
I20260925 10:32:44.093480 5986 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:44.093622 5986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:44.093942 5986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:44.096946 5986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "6109250fd19849c2a67b1bdb4d940394"
format_stamp: "Formatted at 2026-09-25 10:32:44 on dist-test-slave-dxdd"
server_key: "fd44314cd4cb097d9b422bc404e2c602"
server_key_iv: "2ba013d7cac6766196241f453907ee2d"
server_key_version: "encryptionkey@0"
I20260925 10:32:44.097740 5986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "6109250fd19849c2a67b1bdb4d940394"
format_stamp: "Formatted at 2026-09-25 10:32:44 on dist-test-slave-dxdd"
server_key: "fd44314cd4cb097d9b422bc404e2c602"
server_key_iv: "2ba013d7cac6766196241f453907ee2d"
server_key_version: "encryptionkey@0"
I20260925 10:32:44.102993 5986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 10:32:44.106554 6001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:44.107872 5986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:32:44.108049 5986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "6109250fd19849c2a67b1bdb4d940394"
format_stamp: "Formatted at 2026-09-25 10:32:44 on dist-test-slave-dxdd"
server_key: "fd44314cd4cb097d9b422bc404e2c602"
server_key_iv: "2ba013d7cac6766196241f453907ee2d"
server_key_version: "encryptionkey@0"
I20260925 10:32:44.108249 5986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:44.119263 5986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:44.122992 5986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:44.123277 5986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:44.141053 5986 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41877
I20260925 10:32:44.141083 6052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41877 every 8 connection(s)
I20260925 10:32:44.142428 5986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:32:44.145892 6053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:44.150255 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 5986
I20260925 10:32:44.150470 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:32:44.151291 388 external_mini_cluster.cc:1428] Setting key d76e1b66fee12357b16801ee2ec8ec28
I20260925 10:32:44.157069 6053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394: Bootstrap starting.
I20260925 10:32:44.160504 6053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:44.161701 6053 log.cc:824] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:44.164973 6053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394: No bootstrap required, opened a new log
Sep 25 10:32:44 dist-test-slave-dxdd krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:44.173033 6053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6109250fd19849c2a67b1bdb4d940394" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } }
I20260925 10:32:44.173399 6053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:44.173501 6053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6109250fd19849c2a67b1bdb4d940394, State: Initialized, Role: FOLLOWER
I20260925 10:32:44.173898 6056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:44.153185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.0.97.62:41877)
0925 10:32:44.153947 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:44.153958 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:44.154002 (+ 44us) server_negotiation.cc:407] Connection header received
0925 10:32:44.155756 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:44.155778 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:44.156118 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:44.156421 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:44.158092 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:44.159451 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:44.161193 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:44.161991 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:44.162152 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:44.166773 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:44.166781 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:44.166795 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:44.166832 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:44.169837 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:44.170574 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:44.170599 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:44.170605 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:44.170772 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:44.171146 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:44.171151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:44.171156 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:44.171674 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:44.172003 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:44.172432 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 10:32:44.172815 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":358,"threads_started":1}
I20260925 10:32:44.174232 6053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6109250fd19849c2a67b1bdb4d940394" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } }
I20260925 10:32:44.174530 6053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:32:44.174686 6053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:32:44.174871 6053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:44.177798 6053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6109250fd19849c2a67b1bdb4d940394" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } }
I20260925 10:32:44.178253 6053 leader_election.cc:302] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6109250fd19849c2a67b1bdb4d940394; no voters:
I20260925 10:32:44.178884 6053 leader_election.cc:288] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:32:44.178994 6058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:44.179756 6058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [term 1 LEADER]: Becoming Leader. State: Replica: 6109250fd19849c2a67b1bdb4d940394, State: Running, Role: LEADER
I20260925 10:32:44.180292 6058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6109250fd19849c2a67b1bdb4d940394" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } }
I20260925 10:32:44.180879 6053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:44.183477 6060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6109250fd19849c2a67b1bdb4d940394. Latest consensus state: current_term: 1 leader_uuid: "6109250fd19849c2a67b1bdb4d940394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6109250fd19849c2a67b1bdb4d940394" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } } }
I20260925 10:32:44.183828 6060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:44.186538 6059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6109250fd19849c2a67b1bdb4d940394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6109250fd19849c2a67b1bdb4d940394" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } } }
I20260925 10:32:44.186908 6059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:44.187520 6065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:44.192391 6065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:44.202445 6065 catalog_manager.cc:1347] Generated new cluster ID: 945c9aadb1ba49e09cc3147377bb770e
I20260925 10:32:44.202562 6065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:44.242455 6065 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:32:44.243721 6065 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:44.251425 6065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6109250fd19849c2a67b1bdb4d940394: Generated new TSK 0
I20260925 10:32:44.252393 6065 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:32:44.313402 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41877
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:44.631260 6079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:44.631592 6079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:44.631768 6079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:44.639142 6079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:44.639293 6079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:44.639470 6079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:32:44.649143 6079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:44.650559 6079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:44.651853 6079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:44.665025 6084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:44.665093 6085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:44.665086 6087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:44.666128 6079 server_base.cc:1034] running on GCE node
I20260925 10:32:44.666780 6079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:44.667651 6079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:44.668866 6079 hybrid_clock.cc:648] HybridClock initialized: now 1790332364668825 us; error 71 us; skew 500 ppm
I20260925 10:32:44.669184 6079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:44 dist-test-slave-dxdd krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:44.672433 6079 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:32:44.674633 6079 webserver.cc:460] Webserver started at http://127.0.97.1:34099/ using document root <none> and password file <none>
I20260925 10:32:44.675318 6079 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:44.675504 6079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:44.675794 6079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:44.678606 6079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "ee9c58d9773e434c8d43a61cd3ae543b"
format_stamp: "Formatted at 2026-09-25 10:32:44 on dist-test-slave-dxdd"
server_key: "05db8e3ce73b69db2d74c236bfca492d"
server_key_iv: "c86e32312c9d4bae247c8e88db3b39ef"
server_key_version: "encryptionkey@0"
I20260925 10:32:44.679358 6079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ee9c58d9773e434c8d43a61cd3ae543b"
format_stamp: "Formatted at 2026-09-25 10:32:44 on dist-test-slave-dxdd"
server_key: "05db8e3ce73b69db2d74c236bfca492d"
server_key_iv: "c86e32312c9d4bae247c8e88db3b39ef"
server_key_version: "encryptionkey@0"
I20260925 10:32:44.684767 6079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:32:44.688082 6094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:44.689307 6079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:32:44.689482 6079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "ee9c58d9773e434c8d43a61cd3ae543b"
format_stamp: "Formatted at 2026-09-25 10:32:44 on dist-test-slave-dxdd"
server_key: "05db8e3ce73b69db2d74c236bfca492d"
server_key_iv: "c86e32312c9d4bae247c8e88db3b39ef"
server_key_version: "encryptionkey@0"
I20260925 10:32:44.689673 6079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:44.727147 6079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:44.731150 6079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:44.731498 6079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:44.732647 6079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:44.734063 6079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:44.734169 6079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:44.734345 6079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:44.734407 6079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 10:32:44.766757 6079 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43721
I20260925 10:32:44.766790 6206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43721 every 8 connection(s)
I20260925 10:32:44.767980 6079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:32:44.773804 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 6079
I20260925 10:32:44.773996 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:32:44.774814 388 external_mini_cluster.cc:1428] Setting key 2ff1a416cd1143f1075ee81c95e06307
Sep 25 10:32:44 dist-test-slave-dxdd krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:44.787614 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:44.770377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40401 (local address 127.0.97.62:41877)
0925 10:32:44.770812 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:44.770816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:44.771789 (+ 973us) server_negotiation.cc:407] Connection header received
0925 10:32:44.773598 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:44.773602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:44.773663 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:44.773753 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:44.775706 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:44.776478 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:44.777906 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:44.778289 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:44.778471 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:44.782043 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:44.782050 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:44.782056 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:44.782097 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:44.784643 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:44.785521 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:44.785525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:44.785527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:44.785631 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:44.786074 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:44.786077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:44.786079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:44.786360 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:44.786479 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:44.787248 (+ 769us) server_negotiation.cc:299] Negotiation successful
0925 10:32:44.787391 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":196,"threads_started":1}
I20260925 10:32:44.789135 6210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:44.770954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41877 (local address 127.0.97.1:40401)
0925 10:32:44.771624 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:44.771663 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:44.773274 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:44.774036 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:44.774047 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:44.774760 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:44.775539 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:44.775558 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:44.776616 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:44.776621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:44.777764 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:44.777775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:44.778548 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:44.778556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:44.778770 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:44.779417 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:44.779437 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:44.781795 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:44.784775 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:44.784786 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:44.784803 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:44.785413 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:44.785732 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:44.785740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:44.785746 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:44.785974 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:44.786490 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:44.786500 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:44.786895 (+ 395us) client_negotiation.cc:769] Sending connection context
0925 10:32:44.787178 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 10:32:44.787702 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":280,"threads_started":1}
I20260925 10:32:44.790757 6207 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41877
I20260925 10:32:44.791061 6207 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:44.791759 6207 heartbeater.cc:511] Master 127.0.97.62:41877 requested a full tablet report, sending...
I20260925 10:32:44.794003 6018 ts_manager.cc:194] Registered new tserver with Master: ee9c58d9773e434c8d43a61cd3ae543b (127.0.97.1:43721)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260925 10:32:44.795498 6018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:32:44.821240 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41877
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:45.136744 6213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:45.137038 6213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:45.137219 6213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:45.144068 6213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:45.144217 6213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:45.144379 6213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:32:45.153532 6213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:45.154712 6213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:45.155932 6213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:45.168453 6218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:45.168429 6219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:45.168422 6221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:45.169332 6213 server_base.cc:1034] running on GCE node
I20260925 10:32:45.169919 6213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:45.170902 6213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:45.172209 6213 hybrid_clock.cc:648] HybridClock initialized: now 1790332365172141 us; error 51 us; skew 500 ppm
I20260925 10:32:45.172544 6213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:45 dist-test-slave-dxdd krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:45.175638 6213 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:32:45.177021 6213 webserver.cc:460] Webserver started at http://127.0.97.2:32775/ using document root <none> and password file <none>
I20260925 10:32:45.177677 6213 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:45.177824 6213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:45.178117 6213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:45.181079 6213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "40d161df8f364bb2bd76f67e61e64423"
format_stamp: "Formatted at 2026-09-25 10:32:45 on dist-test-slave-dxdd"
server_key: "080eed807df521c728108af41be0f67f"
server_key_iv: "79f957a61bf878873f193e482a8fa16e"
server_key_version: "encryptionkey@0"
I20260925 10:32:45.181864 6213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "40d161df8f364bb2bd76f67e61e64423"
format_stamp: "Formatted at 2026-09-25 10:32:45 on dist-test-slave-dxdd"
server_key: "080eed807df521c728108af41be0f67f"
server_key_iv: "79f957a61bf878873f193e482a8fa16e"
server_key_version: "encryptionkey@0"
I20260925 10:32:45.186930 6213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 10:32:45.190388 6228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:45.191691 6213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:32:45.191979 6213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "40d161df8f364bb2bd76f67e61e64423"
format_stamp: "Formatted at 2026-09-25 10:32:45 on dist-test-slave-dxdd"
server_key: "080eed807df521c728108af41be0f67f"
server_key_iv: "79f957a61bf878873f193e482a8fa16e"
server_key_version: "encryptionkey@0"
I20260925 10:32:45.192225 6213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:45.217098 6213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:45.220945 6213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:45.221230 6213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:45.222163 6213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:45.223425 6213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:45.223558 6213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:45.223716 6213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:45.223862 6213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:45.255661 6213 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34201
I20260925 10:32:45.255744 6340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34201 every 8 connection(s)
I20260925 10:32:45.256878 6213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:32:45.259516 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 6213
I20260925 10:32:45.259670 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:32:45.260092 388 external_mini_cluster.cc:1428] Setting key 2224c7aa57df0bed023aa0de31cadc55
Sep 25 10:32:45 dist-test-slave-dxdd krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260925 10:32:45.283445 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:45.259355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41131 (local address 127.0.97.62:41877)
0925 10:32:45.259569 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:45.259574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:45.260873 (+ 1299us) server_negotiation.cc:407] Connection header received
0925 10:32:45.262696 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:45.262703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:45.262786 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:45.262889 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:45.264192 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:45.265324 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:45.266888 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:45.267630 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:45.267771 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:45.277399 (+ 9628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:45.277404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:45.277409 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:45.277439 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:45.280137 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:45.280903 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:45.280918 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:45.280920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:45.281035 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:45.281562 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:45.281570 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:45.281576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:45.281885 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:45.282046 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:45.282878 (+ 832us) server_negotiation.cc:299] Negotiation successful
0925 10:32:45.283094 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 10:32:45.284446 6343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:45.259901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41877 (local address 127.0.97.2:41131)
0925 10:32:45.260695 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:45.260740 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:45.262443 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:45.263065 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:45.263073 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:45.263521 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:45.264040 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:45.264054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:45.265467 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:45.265470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:45.266745 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:45.266756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:45.267861 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:45.267867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:45.268082 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:45.274719 (+ 6637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:45.274753 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:45.277167 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:45.280281 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:45.280290 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:45.280306 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:45.280736 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:45.281217 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:45.281240 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:45.281242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:45.281414 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:45.282022 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:45.282030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:45.282521 (+ 491us) client_negotiation.cc:769] Sending connection context
0925 10:32:45.282789 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 10:32:45.283310 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":346,"threads_started":1}
I20260925 10:32:45.285949 6341 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41877
I20260925 10:32:45.286284 6341 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:45.287048 6341 heartbeater.cc:511] Master 127.0.97.62:41877 requested a full tablet report, sending...
I20260925 10:32:45.288533 6018 ts_manager.cc:194] Registered new tserver with Master: 40d161df8f364bb2bd76f67e61e64423 (127.0.97.2:34201)
I20260925 10:32:45.289834 6018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:32:45.306864 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41877
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:45.621742 6346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:45.622012 6346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:45.622120 6346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:45.629076 6346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:45.629169 6346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:45.629272 6346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:32:45.638510 6346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:45.639612 6346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:45.640854 6346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:45.653868 6352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:45.653908 6354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:45.654317 6351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:45.654824 6346 server_base.cc:1034] running on GCE node
I20260925 10:32:45.655474 6346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:45.656395 6346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:45.657675 6346 hybrid_clock.cc:648] HybridClock initialized: now 1790332365657581 us; error 46 us; skew 500 ppm
I20260925 10:32:45.657989 6346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:45 dist-test-slave-dxdd krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:45.661054 6346 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:32:45.662523 6346 webserver.cc:460] Webserver started at http://127.0.97.3:35033/ using document root <none> and password file <none>
I20260925 10:32:45.663201 6346 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:45.663342 6346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:45.663671 6346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:45.667905 6346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "d1812317cddb4e60a30038610dc23a8c"
format_stamp: "Formatted at 2026-09-25 10:32:45 on dist-test-slave-dxdd"
server_key: "32a82a9f514cdf25ab45e17937310caa"
server_key_iv: "13157edec6ca32372112144b6a84ab1b"
server_key_version: "encryptionkey@0"
I20260925 10:32:45.668963 6346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d1812317cddb4e60a30038610dc23a8c"
format_stamp: "Formatted at 2026-09-25 10:32:45 on dist-test-slave-dxdd"
server_key: "32a82a9f514cdf25ab45e17937310caa"
server_key_iv: "13157edec6ca32372112144b6a84ab1b"
server_key_version: "encryptionkey@0"
I20260925 10:32:45.674146 6346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 10:32:45.677706 6361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:45.679361 6346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:32:45.679538 6346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "d1812317cddb4e60a30038610dc23a8c"
format_stamp: "Formatted at 2026-09-25 10:32:45 on dist-test-slave-dxdd"
server_key: "32a82a9f514cdf25ab45e17937310caa"
server_key_iv: "13157edec6ca32372112144b6a84ab1b"
server_key_version: "encryptionkey@0"
I20260925 10:32:45.679776 6346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:45.720284 6346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:45.724346 6346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:45.724629 6346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:45.725587 6346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:45.726945 6346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:45.727035 6346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:45.727206 6346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:45.727257 6346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:45.760102 6346 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43579
I20260925 10:32:45.760176 6473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43579 every 8 connection(s)
I20260925 10:32:45.761338 6346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:32:45.765625 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 6346
I20260925 10:32:45.765795 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:32:45.766366 388 external_mini_cluster.cc:1428] Setting key 188200b57b66f50f816fcb531d1b2680
Sep 25 10:32:45 dist-test-slave-dxdd krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:32:45.781000 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:45.763845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43095 (local address 127.0.97.62:41877)
0925 10:32:45.764028 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:45.764032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:45.765144 (+ 1112us) server_negotiation.cc:407] Connection header received
0925 10:32:45.767583 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:45.767588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:45.767675 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:45.767774 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:45.769489 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:45.770343 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:45.771819 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:45.772203 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:45.772332 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:45.775590 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:45.775603 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:45.775607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:45.775637 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:45.777986 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:45.778717 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:45.778725 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:45.778728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:45.778857 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:45.779265 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:45.779270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:45.779274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:45.779529 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:45.779678 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:45.780463 (+ 785us) server_negotiation.cc:299] Negotiation successful
0925 10:32:45.780667 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:32:45.781984 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:45.764309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41877 (local address 127.0.97.3:43095)
0925 10:32:45.764973 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:45.765008 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:45.767300 (+ 2292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:45.768000 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:45.768008 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:45.768692 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:45.769329 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:45.769353 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:45.770483 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:45.770488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:45.771681 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:45.771692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:45.772404 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:45.772413 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:45.772652 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:45.773251 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:45.773289 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:45.775401 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:45.778120 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:45.778129 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:45.778144 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:45.778603 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:45.778974 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:45.778980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:45.778984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:45.779150 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:45.779652 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:45.779659 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:45.779969 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 10:32:45.780328 (+ 359us) client_negotiation.cc:240] Negotiation successful
0925 10:32:45.780802 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":270,"threads_started":1}
I20260925 10:32:45.783319 6474 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41877
I20260925 10:32:45.783632 6474 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:45.784351 6474 heartbeater.cc:511] Master 127.0.97.62:41877 requested a full tablet report, sending...
I20260925 10:32:45.785720 6018 ts_manager.cc:194] Registered new tserver with Master: d1812317cddb4e60a30038610dc23a8c (127.0.97.3:43579)
I20260925 10:32:45.786554 6018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43095
I20260925 10:32:45.793833 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:32:45.801608 6207 heartbeater.cc:503] Master 127.0.97.62:41877 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/test-admin.keytab.
Sep 25 10:32:45 dist-test-slave-dxdd krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:45.823439 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 10:32:45.828464 6481 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:32:45.825976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50242 (local address 127.0.97.1:43721)
0925 10:32:45.826684 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:45.826692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:45.826715 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:32:45.826777 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:45.826783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:45.826978 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:45.827149 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:45.828028 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50242: BlockingRecv error: recv got EOF from 127.0.0.1:50242 (error 108)
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":389,"threads_started":1}
W20260925 10:32:45.828594 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/krb5cc' of type FILE
Sep 25 10:32:45 dist-test-slave-dxdd krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:45.830067 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:32:45 dist-test-slave-dxdd krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:45.832027 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:32:45 dist-test-slave-dxdd krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:32:45.846568 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:45.833700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.0.97.1:43721)
0925 10:32:45.833884 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:45.833889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:45.834073 (+ 184us) server_negotiation.cc:407] Connection header received
0925 10:32:45.834274 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:45.834278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:45.834384 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:45.834487 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:45.835431 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:45.836350 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:45.837544 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:45.838077 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:45.838203 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:45.841495 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:45.841513 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:45.841527 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:45.841561 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:45.844001 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:45.844591 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:45.844602 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:45.844604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:45.844710 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:45.845055 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:45.845061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:45.845064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:45.845399 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:45.845680 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:45.846012 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 10:32:45.846245 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 10:32:45.851996 6481 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:32:45.850238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50272 (local address 127.0.97.1:43721)
0925 10:32:45.850432 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:45.850436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:45.850778 (+ 342us) server_negotiation.cc:407] Connection header received
0925 10:32:45.850996 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:45.850999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:45.851049 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:45.851156 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:45.851768 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50272: BlockingRecv error: recv got EOF from 127.0.0.1:50272 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 10:32:45.852209 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/krb5cc' of type FILE
Sep 25 10:32:45 dist-test-slave-dxdd krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:45.853579 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:32:45 dist-test-slave-dxdd krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:45.854826 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:32:45 dist-test-slave-dxdd krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:32:45.867551 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:45.855994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50284 (local address 127.0.97.1:43721)
0925 10:32:45.856127 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:45.856131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:45.856490 (+ 359us) server_negotiation.cc:407] Connection header received
0925 10:32:45.856568 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:45.856571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:45.856618 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:45.856688 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:45.857492 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:45.858137 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:45.859290 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:45.859698 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:45.859783 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:45.862897 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:45.862901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:45.862904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:45.862930 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:45.865067 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:45.865734 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:45.865741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:45.865744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:45.865868 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:45.866229 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:45.866236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:45.866239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:45.866560 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:45.866818 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:45.867110 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 10:32:45.867290 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 10:32:45.871294 6481 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:32:45.869776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.0.97.1:43721)
0925 10:32:45.869982 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:45.869988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:45.870173 (+ 185us) server_negotiation.cc:407] Connection header received
0925 10:32:45.870275 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:45.870278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:45.870383 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:45.870455 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:45.871124 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50298: BlockingRecv error: recv got EOF from 127.0.0.1:50298 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260925 10:32:45.871605 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332315635355-388-0/krb5kdc/krb5cc' of type FILE
Sep 25 10:32:45 dist-test-slave-dxdd krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:45.872968 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:32:45 dist-test-slave-dxdd krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:45.874255 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:32:45 dist-test-slave-dxdd krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:32:45.887295 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:45.875477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50302 (local address 127.0.97.1:43721)
0925 10:32:45.875651 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:45.875659 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:45.875793 (+ 134us) server_negotiation.cc:407] Connection header received
0925 10:32:45.875896 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:45.875899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:45.875944 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:45.876023 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:45.876910 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:45.877564 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:45.878874 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:45.879261 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:45.879348 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:45.882607 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:45.882611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:45.882614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:45.882639 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:45.885065 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:45.885674 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:45.885678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:45.885680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:45.885811 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:45.886125 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:45.886128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:45.886130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:45.886408 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:45.886636 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:45.886811 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 10:32:45.887014 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:32:45.889063 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 6079
I20260925 10:32:45.901687 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:45.889755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50310 (local address 127.0.97.1:43721)
0925 10:32:45.889895 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:45.889899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:45.890220 (+ 321us) server_negotiation.cc:407] Connection header received
0925 10:32:45.890375 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:45.890378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:45.890429 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:45.890503 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:45.891327 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:45.891955 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:45.893315 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:45.893770 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:45.893941 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:45.896681 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:45.896686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:45.896689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:45.896714 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:45.899652 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:45.900284 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:45.900288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:45.900290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:45.900416 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:45.900708 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:45.900711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:45.900713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:45.900932 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:45.901111 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:45.901315 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 10:32:45.901452 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:32:45.902184 6202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:45.983776 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 6079
I20260925 10:32:46.000778 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 6213
Sep 25 10:32:46 dist-test-slave-dxdd krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:32:46.015770 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:46.001594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.0.97.2:34201)
0925 10:32:46.002089 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:46.002096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:46.002118 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:32:46.002174 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:46.002181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:46.002437 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:46.002606 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:46.003538 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:46.004536 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:46.005769 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:46.006425 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:46.006560 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:46.010283 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:46.010294 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:46.010371 (+ 77us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:46.010426 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:46.013291 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:46.013935 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:46.013942 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:46.013944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:46.014057 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:46.014436 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:46.014439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:46.014441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:46.014743 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:46.014970 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:46.015292 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 10:32:46.015483 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":236,"threads_started":1}
I20260925 10:32:46.016971 6336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:46.099136 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 6213
I20260925 10:32:46.114626 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 6346
Sep 25 10:32:46 dist-test-slave-dxdd krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:32:46.129920 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:46.115590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46012 (local address 127.0.97.3:43579)
0925 10:32:46.116193 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:46.116200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:46.116228 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:32:46.116291 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:46.116297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:46.116503 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:46.116684 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:46.117749 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:46.118838 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:46.120302 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:46.120875 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:46.121024 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:46.124947 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:46.124958 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:46.124972 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:46.125002 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:46.127607 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:46.128202 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:46.128209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:46.128211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:46.128338 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:46.128660 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:46.128665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:46.128667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:46.129033 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:46.129270 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:46.129462 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 10:32:46.129642 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":308,"threads_started":1}
I20260925 10:32:46.131441 6469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:46.212366 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 6346
I20260925 10:32:46.228127 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 5986
I20260925 10:32:46.229387 6048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:46.327314 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 5986
2026-09-25T10:32:46Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3875 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:32:46 dist-test-slave-dxdd krb5kdc[6501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:32:46 dist-test-slave-dxdd krb5kdc[6501](info): set up 2 sockets
Sep 25 10:32:46 dist-test-slave-dxdd krb5kdc[6501](info): commencing operation
krb5kdc: starting...
W20260925 10:32:47.395035 388 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:32:47 dist-test-slave-dxdd krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:32:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:32:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260925 10:32:47.586229 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46413
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:47.899302 6514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:47.899569 6514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:47.899753 6514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:47.906780 6514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:47.906877 6514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:47.906971 6514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:47.907085 6514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:47.907142 6514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:32:47.916514 6514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:47.917769 6514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:47.919111 6514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:47.930522 6519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:47.930544 6520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:47.930595 6522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:47.931339 6514 server_base.cc:1034] running on GCE node
I20260925 10:32:47.932128 6514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:47.933482 6514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:47.934702 6514 hybrid_clock.cc:648] HybridClock initialized: now 1790332367934641 us; error 45 us; skew 500 ppm
I20260925 10:32:47.935034 6514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:47 dist-test-slave-dxdd krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332367, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:47.937956 6514 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260925 10:32:47.939417 6514 webserver.cc:460] Webserver started at http://127.0.97.62:43537/ using document root <none> and password file <none>
I20260925 10:32:47.940073 6514 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:47.940222 6514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:47.940510 6514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:47.943370 6514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "e8edbe7ca91c47a2ac5daed766a95286"
format_stamp: "Formatted at 2026-09-25 10:32:47 on dist-test-slave-dxdd"
server_key: "3a6e3522697e065e993c1f657c3a0261"
server_key_iv: "93d1fc79441059b7281ceee21fcf29e6"
server_key_version: "encryptionkey@0"
I20260925 10:32:47.944096 6514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "e8edbe7ca91c47a2ac5daed766a95286"
format_stamp: "Formatted at 2026-09-25 10:32:47 on dist-test-slave-dxdd"
server_key: "3a6e3522697e065e993c1f657c3a0261"
server_key_iv: "93d1fc79441059b7281ceee21fcf29e6"
server_key_version: "encryptionkey@0"
I20260925 10:32:47.948946 6514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 10:32:47.952147 6529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:47.953490 6514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 10:32:47.953657 6514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "e8edbe7ca91c47a2ac5daed766a95286"
format_stamp: "Formatted at 2026-09-25 10:32:47 on dist-test-slave-dxdd"
server_key: "3a6e3522697e065e993c1f657c3a0261"
server_key_iv: "93d1fc79441059b7281ceee21fcf29e6"
server_key_version: "encryptionkey@0"
I20260925 10:32:47.953837 6514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:48.012961 6514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:48.016829 6514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:48.017124 6514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:48.035542 6514 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46413
I20260925 10:32:48.035569 6580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46413 every 8 connection(s)
I20260925 10:32:48.036962 6514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:32:48.040498 6581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:48.044262 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 6514
I20260925 10:32:48.044490 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:32:48.045126 388 external_mini_cluster.cc:1428] Setting key 10441f0843542c74b316354f5610284b
I20260925 10:32:48.051049 6581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286: Bootstrap starting.
I20260925 10:32:48.054448 6581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:48.055541 6581 log.cc:824] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:48.058899 6581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286: No bootstrap required, opened a new log
Sep 25 10:32:48 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260925 10:32:48.068976 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:48.047018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33158 (local address 127.0.97.62:46413)
0925 10:32:48.047744 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:48.047755 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:48.047796 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:32:48.049565 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:48.049586 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:48.049943 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:48.050261 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:48.052033 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:48.053308 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:48.055119 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:48.055850 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:48.056001 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:48.060689 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:48.060698 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:48.060716 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:48.060761 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:48.064656 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:48.065337 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:48.065357 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:48.065368 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:48.065554 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:48.065906 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:48.065911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:48.065915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:48.066618 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:48.066888 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:48.067244 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 10:32:48.067668 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":344,"threads_started":1}
I20260925 10:32:48.069531 6581 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } }
I20260925 10:32:48.069957 6581 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:48.070073 6581 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8edbe7ca91c47a2ac5daed766a95286, State: Initialized, Role: FOLLOWER
I20260925 10:32:48.070665 6581 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } }
I20260925 10:32:48.070897 6581 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:32:48.071024 6581 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:32:48.071240 6581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:48.073648 6581 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } }
I20260925 10:32:48.074124 6581 leader_election.cc:302] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8edbe7ca91c47a2ac5daed766a95286; no voters:
I20260925 10:32:48.074685 6581 leader_election.cc:288] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:32:48.074831 6586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:48.075500 6586 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [term 1 LEADER]: Becoming Leader. State: Replica: e8edbe7ca91c47a2ac5daed766a95286, State: Running, Role: LEADER
I20260925 10:32:48.076009 6586 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } }
I20260925 10:32:48.076607 6581 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:48.078682 6587 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } }
I20260925 10:32:48.079104 6587 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:48.079814 6595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:48.080904 6588 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8edbe7ca91c47a2ac5daed766a95286. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } }
I20260925 10:32:48.082460 6588 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:48.091079 6595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:48.099397 6595 catalog_manager.cc:1347] Generated new cluster ID: 04db75e16edb4e8fa187793f7336f08a
I20260925 10:32:48.099493 6595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:48.113631 6595 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:32:48.115197 6595 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:48.120784 6595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286: Generated new TSK 0
I20260925 10:32:48.121500 6595 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260925 10:32:48.173530 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46413
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:48.498713 6607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:48.499010 6607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:48.499177 6607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:48.506074 6607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:48.506217 6607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:48.506410 6607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:32:48.515924 6607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:48.517122 6607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:48.518404 6607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:48.531124 6615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:48.531196 6612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:48.531163 6613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:48.532163 6607 server_base.cc:1034] running on GCE node
I20260925 10:32:48.532702 6607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:48.533583 6607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:48.534799 6607 hybrid_clock.cc:648] HybridClock initialized: now 1790332368534749 us; error 42 us; skew 500 ppm
I20260925 10:32:48.535076 6607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:48 dist-test-slave-dxdd krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332368, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:48.538005 6607 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260925 10:32:48.539402 6607 webserver.cc:460] Webserver started at http://127.0.97.1:32777/ using document root <none> and password file <none>
I20260925 10:32:48.540014 6607 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:48.540169 6607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:48.540455 6607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:48.543186 6607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "65690ad1c05d4a8083522828aca777ba"
format_stamp: "Formatted at 2026-09-25 10:32:48 on dist-test-slave-dxdd"
server_key: "fa751b10999b0e6574e78669ab40a4a8"
server_key_iv: "4b3747e894739cc4787931bce6df995a"
server_key_version: "encryptionkey@0"
I20260925 10:32:48.543977 6607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "65690ad1c05d4a8083522828aca777ba"
format_stamp: "Formatted at 2026-09-25 10:32:48 on dist-test-slave-dxdd"
server_key: "fa751b10999b0e6574e78669ab40a4a8"
server_key_iv: "4b3747e894739cc4787931bce6df995a"
server_key_version: "encryptionkey@0"
I20260925 10:32:48.548589 6607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:32:48.551719 6622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:48.552836 6607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 10:32:48.553000 6607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "65690ad1c05d4a8083522828aca777ba"
format_stamp: "Formatted at 2026-09-25 10:32:48 on dist-test-slave-dxdd"
server_key: "fa751b10999b0e6574e78669ab40a4a8"
server_key_iv: "4b3747e894739cc4787931bce6df995a"
server_key_version: "encryptionkey@0"
I20260925 10:32:48.553191 6607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:48.573237 6607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:48.577266 6607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:48.577481 6607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:48.578467 6607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:48.579631 6607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:48.579702 6607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:48.579835 6607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:48.579882 6607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:48.610097 6607 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33123
I20260925 10:32:48.610183 6734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33123 every 8 connection(s)
I20260925 10:32:48.611332 6607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:32:48.615091 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 6607
I20260925 10:32:48.615283 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:32:48.615926 388 external_mini_cluster.cc:1428] Setting key d05f313ab3b1244f5ecdac43816a8e82
Sep 25 10:32:48 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332368, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260925 10:32:48.632503 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:48.613675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59977 (local address 127.0.97.62:46413)
0925 10:32:48.614075 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:48.614079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:48.614899 (+ 820us) server_negotiation.cc:407] Connection header received
0925 10:32:48.616834 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:48.616839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:48.616939 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:48.617054 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:48.618669 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:48.619379 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:48.621571 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:48.621950 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:48.622078 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:48.625686 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:48.625694 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:48.625700 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:48.625743 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:48.629107 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:48.629999 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:48.630007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:48.630014 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:48.630156 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:48.630605 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:48.630613 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:48.630620 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:48.631007 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:48.631157 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:48.631825 (+ 668us) server_negotiation.cc:299] Negotiation successful
0925 10:32:48.632072 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":219,"threads_started":1}
I20260925 10:32:48.633126 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:48.614059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.1:59977)
0925 10:32:48.614747 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:48.614786 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:48.616556 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:48.617283 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:48.617291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:48.617914 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:48.618507 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:48.618526 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:48.619533 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:48.619538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:48.621429 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:48.621441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:48.622167 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:48.622173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:48.622444 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:48.623197 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:48.623218 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:48.625463 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:48.629312 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:48.629324 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:48.629342 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:48.629865 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:48.630259 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:48.630273 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:48.630276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:48.630482 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:48.631123 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:48.631129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:48.631438 (+ 309us) client_negotiation.cc:769] Sending connection context
0925 10:32:48.631687 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 10:32:48.632077 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":307,"threads_started":1}
I20260925 10:32:48.634671 6735 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46413
I20260925 10:32:48.635000 6735 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:48.635870 6735 heartbeater.cc:511] Master 127.0.97.62:46413 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260925 10:32:48.637897 6546 ts_manager.cc:194] Registered new tserver with Master: 65690ad1c05d4a8083522828aca777ba (127.0.97.1:33123)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260925 10:32:48.639415 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260925 10:32:48.661762 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46413
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:48.972805 6741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:48.973057 6741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:48.973213 6741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:48.980679 6741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:48.980768 6741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:48.980954 6741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:32:48.989887 6741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:48.991041 6741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:48.992369 6741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:49.004659 6747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:49.004748 6746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:49.004779 6749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:49.005702 6741 server_base.cc:1034] running on GCE node
I20260925 10:32:49.006278 6741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:49.007189 6741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:49.008404 6741 hybrid_clock.cc:648] HybridClock initialized: now 1790332369008362 us; error 37 us; skew 500 ppm
I20260925 10:32:49.008693 6741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:49 dist-test-slave-dxdd krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332369, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:49.011623 6741 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260925 10:32:49.012893 6741 webserver.cc:460] Webserver started at http://127.0.97.2:34933/ using document root <none> and password file <none>
I20260925 10:32:49.013515 6741 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:49.013651 6741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:49.013939 6741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:49.016685 6741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "2ba74a37e8f444618ecb68f3c6a44c88"
format_stamp: "Formatted at 2026-09-25 10:32:49 on dist-test-slave-dxdd"
server_key: "6677dfdb0d5ce0742f615636234e9742"
server_key_iv: "b9dffdb262f8cda32da9108c0cb281af"
server_key_version: "encryptionkey@0"
I20260925 10:32:49.017405 6741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2ba74a37e8f444618ecb68f3c6a44c88"
format_stamp: "Formatted at 2026-09-25 10:32:49 on dist-test-slave-dxdd"
server_key: "6677dfdb0d5ce0742f615636234e9742"
server_key_iv: "b9dffdb262f8cda32da9108c0cb281af"
server_key_version: "encryptionkey@0"
I20260925 10:32:49.022102 6741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 10:32:49.025333 6756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:49.026624 6741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:32:49.026793 6741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "2ba74a37e8f444618ecb68f3c6a44c88"
format_stamp: "Formatted at 2026-09-25 10:32:49 on dist-test-slave-dxdd"
server_key: "6677dfdb0d5ce0742f615636234e9742"
server_key_iv: "b9dffdb262f8cda32da9108c0cb281af"
server_key_version: "encryptionkey@0"
I20260925 10:32:49.026986 6741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:49.057834 6741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:49.061749 6741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:49.062021 6741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:49.062951 6741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:49.064178 6741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:49.064282 6741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:49.064451 6741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:49.064530 6741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:49.096046 6741 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33031
I20260925 10:32:49.096127 6868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33031 every 8 connection(s)
I20260925 10:32:49.097239 6741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:32:49.099268 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 6741
I20260925 10:32:49.099442 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:32:49.099998 388 external_mini_cluster.cc:1428] Setting key 4c5df5f12776ca5e054b7c1c0964bd68
Sep 25 10:32:49 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332369, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260925 10:32:49.120290 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:49.099903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50021 (local address 127.0.97.62:46413)
0925 10:32:49.100113 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:49.100120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:49.101380 (+ 1260us) server_negotiation.cc:407] Connection header received
0925 10:32:49.103904 (+ 2524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:49.103907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:49.103967 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:49.104049 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:49.105984 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:49.106776 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:49.109110 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:49.109796 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:49.109946 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:49.113449 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:49.113457 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:49.113464 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:49.113504 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:49.117059 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:49.117917 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:49.117924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:49.117928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:49.118092 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:49.118535 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:49.118543 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:49.118549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:49.118909 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:49.119083 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:49.119698 (+ 615us) server_negotiation.cc:299] Negotiation successful
0925 10:32:49.119932 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:32:49.120960 6871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:49.100444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.2:50021)
0925 10:32:49.101185 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:49.101250 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:49.103596 (+ 2346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:49.104313 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:49.104327 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:49.105060 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:49.105812 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:49.105839 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:49.106911 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:49.106927 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:49.108967 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:49.108978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:49.109993 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:49.109997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:49.110172 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:49.110937 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:49.110957 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:49.113240 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:49.117268 (+ 4028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:49.117275 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:49.117285 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:49.117800 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:49.118198 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:49.118201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:49.118204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:49.118425 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:49.119045 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:49.119050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:49.119311 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 10:32:49.119522 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 10:32:49.119868 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":309,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260925 10:32:49.122483 6869 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46413
I20260925 10:32:49.122774 6869 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:49.123431 6869 heartbeater.cc:511] Master 127.0.97.62:46413 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260925 10:32:49.124991 6546 ts_manager.cc:194] Registered new tserver with Master: 2ba74a37e8f444618ecb68f3c6a44c88 (127.0.97.2:33031)
I20260925 10:32:49.126173 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260925 10:32:49.147552 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46413
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:49.450479 6874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:49.450729 6874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:49.450907 6874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:49.457953 6874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:49.458072 6874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:49.458246 6874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:32:49.467885 6874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:49.469120 6874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:49.470384 6874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:49.483269 6880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:49.483345 6879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:49.483210 6882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:49.484043 6874 server_base.cc:1034] running on GCE node
I20260925 10:32:49.484576 6874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:49.485473 6874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:49.486724 6874 hybrid_clock.cc:648] HybridClock initialized: now 1790332369486665 us; error 49 us; skew 500 ppm
I20260925 10:32:49.487030 6874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:49 dist-test-slave-dxdd krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332369, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:49.489950 6874 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260925 10:32:49.491413 6874 webserver.cc:460] Webserver started at http://127.0.97.3:37817/ using document root <none> and password file <none>
I20260925 10:32:49.492081 6874 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:49.492224 6874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:49.492527 6874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:49.495391 6874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "9ba24b6ac5a349b3b3a88387a76dba5c"
format_stamp: "Formatted at 2026-09-25 10:32:49 on dist-test-slave-dxdd"
server_key: "e1ac8a5f7e3dbb6ba0b4be67bd00af8f"
server_key_iv: "07ed818589754b3acf6ab66b5ddccfc5"
server_key_version: "encryptionkey@0"
I20260925 10:32:49.496146 6874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9ba24b6ac5a349b3b3a88387a76dba5c"
format_stamp: "Formatted at 2026-09-25 10:32:49 on dist-test-slave-dxdd"
server_key: "e1ac8a5f7e3dbb6ba0b4be67bd00af8f"
server_key_iv: "07ed818589754b3acf6ab66b5ddccfc5"
server_key_version: "encryptionkey@0"
I20260925 10:32:49.500962 6874 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 10:32:49.504395 6889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:49.505625 6874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:32:49.505793 6874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "9ba24b6ac5a349b3b3a88387a76dba5c"
format_stamp: "Formatted at 2026-09-25 10:32:49 on dist-test-slave-dxdd"
server_key: "e1ac8a5f7e3dbb6ba0b4be67bd00af8f"
server_key_iv: "07ed818589754b3acf6ab66b5ddccfc5"
server_key_version: "encryptionkey@0"
I20260925 10:32:49.505998 6874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:49.519557 6874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:49.523343 6874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:49.523622 6874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:49.524596 6874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:49.525820 6874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:49.525924 6874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:49.526084 6874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:49.526163 6874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:49.558624 6874 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37187
I20260925 10:32:49.558665 7001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37187 every 8 connection(s)
I20260925 10:32:49.559908 6874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:32:49.564848 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 6874
I20260925 10:32:49.565022 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:32:49.565433 388 external_mini_cluster.cc:1428] Setting key cb86a075541791418a9e944d972a85a5
Sep 25 10:32:49 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332369, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260925 10:32:49.578969 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:49.562438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57617 (local address 127.0.97.62:46413)
0925 10:32:49.562600 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:49.562605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:49.563717 (+ 1112us) server_negotiation.cc:407] Connection header received
0925 10:32:49.565555 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:49.565561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:49.565644 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:49.565748 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:49.567487 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:49.568427 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:49.569877 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:49.570270 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:49.570451 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:49.573630 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:49.573640 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:49.573643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:49.573672 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:49.576188 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:49.576841 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:49.576848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:49.576851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:49.576972 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:49.577370 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:49.577376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:49.577379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:49.577625 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:49.577769 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:49.578407 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 10:32:49.578609 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:32:49.579954 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:49.562901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.3:57617)
0925 10:32:49.563576 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:49.563610 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:49.565305 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:49.565980 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:49.565987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:49.566705 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:49.567327 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:49.567354 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:49.568567 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:49.568572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:49.569741 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:49.569751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:49.570534 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:49.570543 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:49.570767 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:49.571424 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:49.571445 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:49.573440 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:49.576326 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:49.576334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:49.576348 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:49.576726 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:49.577089 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:49.577093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:49.577097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:49.577265 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:49.577741 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:49.577748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:49.578054 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 10:32:49.578303 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 10:32:49.578805 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":274,"threads_started":1}
I20260925 10:32:49.581115 7002 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46413
I20260925 10:32:49.581382 7002 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:49.582043 7002 heartbeater.cc:511] Master 127.0.97.62:46413 requested a full tablet report, sending...
I20260925 10:32:49.583304 6546 ts_manager.cc:194] Registered new tserver with Master: 9ba24b6ac5a349b3b3a88387a76dba5c (127.0.97.3:37187)
I20260925 10:32:49.584050 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57617
I20260925 10:32:49.592293 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260925 10:32:49.635737 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43321
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46413,127.0.97.61:43321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:32:49.642086 6735 heartbeater.cc:503] Master 127.0.97.62:46413 was elected leader, sending a full tablet report...
W20260925 10:32:49.945580 7007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:49.945823 7007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:49.946000 7007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:49.952950 7007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:49.953081 7007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:49.953163 7007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:49.953265 7007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:49.953344 7007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:32:49.962546 7007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46413,127.0.97.61:43321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
W20260925 10:32:49.962852 7007 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:46413,127.0.97.61:43321'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 10:32:49.963775 7007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:49.965058 7007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:49.975934 7013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:49.975965 7015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:49.976503 7012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:49.977047 7007 server_base.cc:1034] running on GCE node
I20260925 10:32:49.977794 7007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:49.979193 7007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:49.980415 7007 hybrid_clock.cc:648] HybridClock initialized: now 1790332369980351 us; error 42 us; skew 500 ppm
I20260925 10:32:49.980727 7007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:49 dist-test-slave-dxdd krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332369, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:49.983722 7007 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260925 10:32:49.985081 7007 webserver.cc:460] Webserver started at http://127.0.97.61:40133/ using document root <none> and password file <none>
I20260925 10:32:49.985759 7007 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:49.985879 7007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:49.986197 7007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:49.988976 7007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/data/instance:
uuid: "ca8ca5ced019409bacfa9b2e62f58a58"
format_stamp: "Formatted at 2026-09-25 10:32:49 on dist-test-slave-dxdd"
server_key: "088e098e387c21ef4263362bc3b796f3"
server_key_iv: "aaf0b1bff2bdb61cbf1c70a9de583ceb"
server_key_version: "encryptionkey@0"
I20260925 10:32:49.989707 7007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/wal/instance:
uuid: "ca8ca5ced019409bacfa9b2e62f58a58"
format_stamp: "Formatted at 2026-09-25 10:32:49 on dist-test-slave-dxdd"
server_key: "088e098e387c21ef4263362bc3b796f3"
server_key_iv: "aaf0b1bff2bdb61cbf1c70a9de583ceb"
server_key_version: "encryptionkey@0"
I20260925 10:32:49.994366 7007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 10:32:49.997385 7022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:49.998524 7007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260925 10:32:49.998677 7007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/wal
uuid: "ca8ca5ced019409bacfa9b2e62f58a58"
format_stamp: "Formatted at 2026-09-25 10:32:49 on dist-test-slave-dxdd"
server_key: "088e098e387c21ef4263362bc3b796f3"
server_key_iv: "aaf0b1bff2bdb61cbf1c70a9de583ceb"
server_key_version: "encryptionkey@0"
I20260925 10:32:49.998867 7007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:50.016474 7007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:50.020133 7007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:50.020362 7007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:50.038345 7007 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:43321
I20260925 10:32:50.038367 7073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:43321 every 8 connection(s)
I20260925 10:32:50.039619 7007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/data/info.pb
I20260925 10:32:50.042341 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 7007
I20260925 10:32:50.043699 7074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:50.051494 7074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:32:50 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 25 10:32:50 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332369, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260925 10:32:50.065644 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.044542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.0.97.61:43321)
0925 10:32:50.045363 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.045374 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.045441 (+ 67us) server_negotiation.cc:407] Connection header received
0925 10:32:50.046894 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.046915 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.047269 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.047670 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.049045 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.050631 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.051866 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.052574 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.052751 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.057757 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.057767 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.057780 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.057818 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.061432 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.062223 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.062236 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.062241 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.062456 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.062888 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.062892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.062894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.063326 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.063620 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.064007 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.064396 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":327,"threads_started":1}
I20260925 10:32:50.068627 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.053733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33168 (local address 127.0.97.62:46413)
0925 10:32:50.053924 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.053931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.054454 (+ 523us) server_negotiation.cc:407] Connection header received
0925 10:32:50.054759 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.054766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.054867 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.054937 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.056089 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.057000 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.058776 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.059134 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.059224 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.062768 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.062774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.062778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.062816 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.065948 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.066695 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.066700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.066703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.066842 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.067201 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.067206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.067209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.067530 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.067666 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.068100 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.068327 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:32:50.068698 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.053664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:33168)
0925 10:32:50.054334 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:50.054378 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:50.054572 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:50.055198 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:50.055204 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:50.055719 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:50.055926 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.055933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.057125 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.057130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.058642 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.058659 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.059310 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.059319 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.059468 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.059993 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:50.060015 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:50.062605 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:50.066082 (+ 3477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.066090 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.066094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.066571 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.066935 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.066942 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.066945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.067088 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.067661 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:50.067667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:50.067827 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 10:32:50.068010 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 10:32:50.068322 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":301,"threads_started":1}
I20260925 10:32:50.069939 7074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:32:50 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332369, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260925 10:32:50.080878 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.070700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43321 (local address 127.0.0.1:40184)
0925 10:32:50.070872 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:50.070890 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:50.070973 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:50.071159 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:50.071162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:50.071429 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:50.071563 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.071570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.072396 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.072399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.073190 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.073199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.073794 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.073797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.073926 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.074424 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:50.074439 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:50.076330 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:50.079044 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.079048 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.079050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.079457 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.079747 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.079750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.079752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.079877 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.080266 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:50.080269 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:50.080384 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 10:32:50.080500 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 10:32:50.080678 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 10:32:50.081053 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.070650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.0.97.61:43321)
0925 10:32:50.070801 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.070805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.070952 (+ 147us) server_negotiation.cc:407] Connection header received
0925 10:32:50.071035 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.071038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.071082 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.071144 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.071667 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.072300 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.073290 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.073679 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.073758 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.076484 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.076488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.076491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.076516 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.078934 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.079548 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.079551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.079553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.079664 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.079963 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.079967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.079969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.080180 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.080275 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.080633 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.080783 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:32:50.085108 7074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: Bootstrap starting.
I20260925 10:32:50.088012 7074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:50.088948 7074 log.cc:824] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:50.091289 7074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: No bootstrap required, opened a new log
I20260925 10:32:50.099669 7074 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } }
I20260925 10:32:50.100001 7074 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:50.100091 7074 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8ca5ced019409bacfa9b2e62f58a58, State: Initialized, Role: FOLLOWER
I20260925 10:32:50.100662 7074 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } }
I20260925 10:32:50.101580 7080 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } } }
I20260925 10:32:50.101867 7080 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:32:50.102594 7074 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 10:32:50.111091 7091 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:32:50.111263 7091 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:32:50.113338 7007 master_runner.cc:428] Detected that this master ca8ca5ced019409bacfa9b2e62f58a58 is joining an existing cluster
I20260925 10:32:50.113443 7007 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:43321
I20260925 10:32:50.128871 6869 heartbeater.cc:503] Master 127.0.97.62:46413 was elected leader, sending a full tablet report...
I20260925 10:32:50.133970 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.117551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43321 (local address 127.0.0.1:40188)
0925 10:32:50.118208 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:50.118225 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:50.118396 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:50.118876 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:50.118898 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:50.119566 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:50.120056 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.120070 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.121605 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.121609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.123210 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.123224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.124460 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.124466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.124622 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.125291 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:50.125320 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:50.126524 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:50.130830 (+ 4306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.130835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.130839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.131519 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.131881 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.131895 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.131898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.132063 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.132757 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:50.132763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:50.132906 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 10:32:50.133046 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 10:32:50.133372 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"spinlock_wait_cycles":21248,"thread_start_us":395,"threads_started":1}
I20260925 10:32:50.134630 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.117231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33184 (local address 127.0.97.62:46413)
0925 10:32:50.117388 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.117392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.119007 (+ 1615us) server_negotiation.cc:407] Connection header received
0925 10:32:50.119243 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.119251 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.119337 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.119415 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.120830 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.121884 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.123929 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.124579 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.124668 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.126918 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.126924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.126929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.126966 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.130703 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.132110 (+ 1407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.132117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.132120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.132267 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.132704 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.132709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.132714 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.133086 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.133214 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.133794 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.133994 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:32:50.134981 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.117241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.0.97.61:43321)
0925 10:32:50.117413 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.117421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.118325 (+ 904us) server_negotiation.cc:407] Connection header received
0925 10:32:50.118516 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.118523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.118608 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.118711 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.120207 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.121425 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.123386 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.124125 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.124221 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.126728 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.126734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.126738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.126770 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.130693 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.131641 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.131646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.131649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.131759 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.132145 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.132151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.132156 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.132543 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.132706 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.134257 (+ 1551us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.134538 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":58}
I20260925 10:32:50.136132 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.117605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:33184)
0925 10:32:50.118915 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:50.118930 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:50.119088 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:50.119600 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:50.119607 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:50.120307 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:50.120654 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.120666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.122068 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.122074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.123776 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.123789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.124941 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.124945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.125074 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.125618 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:50.125642 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:50.126733 (+ 1091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:50.130830 (+ 4097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.130835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.130839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.131960 (+ 1121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.132404 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.132409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.132413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.132576 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.133290 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:50.133295 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:50.133502 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 10:32:50.135267 (+ 1765us) client_negotiation.cc:240] Negotiation successful
0925 10:32:50.135610 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1196,"mutex_wait_us":36,"thread_start_us":1023,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 25 10:32:50 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332367, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260925 10:32:50.162882 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.142552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.0.97.61:43321)
0925 10:32:50.142711 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.142718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.146453 (+ 3735us) server_negotiation.cc:407] Connection header received
0925 10:32:50.146729 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.146737 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.146829 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.146954 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.147909 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.148747 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.150612 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.151377 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.151566 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.156016 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.156026 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.156031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.156075 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.159573 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.160450 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.160456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.160459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.160604 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.161095 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.161100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.161104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.161450 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.161587 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.162223 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.162503 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:32:50.162955 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.142299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43321 (local address 127.0.0.1:40196)
0925 10:32:50.146262 (+ 3963us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:50.146293 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:50.146535 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:50.146989 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:50.146996 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:50.147571 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:50.147759 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.147767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.148896 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.148901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.150421 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.150435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.151596 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.151602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.151807 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.152762 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:50.152786 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:50.155814 (+ 3028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:50.159715 (+ 3901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.159728 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.159739 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.160311 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.160726 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.160734 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.160740 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.160956 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.161594 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:50.161605 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:50.161838 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 10:32:50.162057 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 10:32:50.162435 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3751,"thread_start_us":3588,"threads_started":1}
I20260925 10:32:50.165303 6546 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:43321: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 43321
}
attrs {
promote: true
}
}
dest_uuid: "e8edbe7ca91c47a2ac5daed766a95286"
cas_config_opid_index: -1
I20260925 10:32:50.166626 6546 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } }
I20260925 10:32:50.169690 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ca8ca5ced019409bacfa9b2e62f58a58 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } } }
I20260925 10:32:50.170754 7049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:50.173801 7104 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:43321
I20260925 10:32:50.174199 7103 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } } }
I20260925 10:32:50.174525 7007 master.cc:537] Master@127.0.97.61:43321 shutting down...
I20260925 10:32:50.174702 7103 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:50.174963 7049 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 FOLLOWER]: Refusing update from remote peer e8edbe7ca91c47a2ac5daed766a95286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260925 10:32:50.176224 7104 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:50.177974 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36757
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46413,127.0.97.61:43321,127.0.97.60:36757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:50.179765 6533 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 10:32:50.182605 6533 consensus_peers.cc:487] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 -> Peer ca8ca5ced019409bacfa9b2e62f58a58 (127.0.97.61:43321): Couldn't send request to peer ca8ca5ced019409bacfa9b2e62f58a58. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 10:32:50.187284 7007 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:50.187654 7007 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:50.187844 7007 tablet_replica.cc:330] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: stopping tablet replica
I20260925 10:32:50.193473 7007 master.cc:552] Master@127.0.97.61:43321 shutdown complete.
I20260925 10:32:50.193568 7007 master_runner.cc:305] Clearing existing system tablet
I20260925 10:32:50.194070 7007 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:50.194159 7007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:50.196362 7007 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:32:50.197767 7109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:50.198076 7007 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:50.198375 7007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:32:50.198524 7007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/wal
uuid: "ca8ca5ced019409bacfa9b2e62f58a58"
format_stamp: "Formatted at 2026-09-25 10:32:49 on dist-test-slave-dxdd"
server_key: "088e098e387c21ef4263362bc3b796f3"
server_key_iv: "aaf0b1bff2bdb61cbf1c70a9de583ceb"
server_key_version: "encryptionkey@0"
I20260925 10:32:50.201857 7007 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:50.207026 7007 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 10:32:50.207213 7007 log.cc:1196] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 10:32:50.207594 7007 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: Deleting consensus metadata
I20260925 10:32:50.207856 7007 master_runner.cc:315] Copying system tablet from 127.0.97.62:46413
I20260925 10:32:50.210384 7007 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46413
I20260925 10:32:50.222914 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.211155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.0.97.62:46413)
0925 10:32:50.211362 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.211372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.211851 (+ 479us) server_negotiation.cc:407] Connection header received
0925 10:32:50.212055 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.212060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.212145 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.212251 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.213131 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.214019 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.215415 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.215818 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.215951 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.217796 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.217803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.217807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.217846 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.220364 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.220989 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.220996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.221000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.221121 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.221519 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.221526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.221529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.221796 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.221949 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.222291 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.222525 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:32:50.222914 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.211141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:33186)
0925 10:32:50.211734 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:50.211758 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:50.211920 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:50.212278 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:50.212283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:50.212745 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:50.212978 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.212987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.214154 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.214160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.215267 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.215278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.215967 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.215973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.216147 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.216906 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:50.216929 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:50.217609 (+ 680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:50.220501 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.220509 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.220513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.220871 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.221237 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.221243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.221246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.221417 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.221904 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:50.221908 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:50.222080 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 10:32:50.222212 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 10:32:50.222494 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":386,"threads_started":1}
I20260925 10:32:50.224085 6566 tablet_copy_service.cc:140] P e8edbe7ca91c47a2ac5daed766a95286: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ca8ca5ced019409bacfa9b2e62f58a58 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:33186)
I20260925 10:32:50.224315 6566 tablet_copy_service.cc:161] P e8edbe7ca91c47a2ac5daed766a95286: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ca8ca5ced019409bacfa9b2e62f58a58 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:33186: session id = ca8ca5ced019409bacfa9b2e62f58a58-00000000000000000000000000000000
I20260925 10:32:50.227283 6566 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 10:32:50.228883 7007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 10:32:50.233850 7007 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: tablet copy: Starting download of 0 data blocks...
I20260925 10:32:50.234133 7007 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: tablet copy: Starting download of 1 WAL segments...
I20260925 10:32:50.236303 7007 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 10:32:50.239214 6566 tablet_copy_service.cc:342] P e8edbe7ca91c47a2ac5daed766a95286: Request end of tablet copy session ca8ca5ced019409bacfa9b2e62f58a58-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:33186
I20260925 10:32:50.239387 6566 tablet_copy_service.cc:433] P e8edbe7ca91c47a2ac5daed766a95286: ending tablet copy session ca8ca5ced019409bacfa9b2e62f58a58-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ca8ca5ced019409bacfa9b2e62f58a58
I20260925 10:32:50.250578 7007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:50.253062 7116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:50.254079 7117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:50.261363 7007 server_base.cc:1034] running on GCE node
W20260925 10:32:50.261690 7120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:50.262122 7007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:50.266527 7007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:50.267755 7007 hybrid_clock.cc:648] HybridClock initialized: now 1790332370267670 us; error 50 us; skew 500 ppm
I20260925 10:32:50.268010 7007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:50 dist-test-slave-dxdd krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332370, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:50.270193 7007 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260925 10:32:50.271323 7007 webserver.cc:460] Webserver started at http://127.0.97.61:46311/ using document root <none> and password file <none>
I20260925 10:32:50.271680 7007 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:50.271763 7007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:50.274808 7007 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 10:32:50.276777 7126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:50.277272 7007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:32:50.277421 7007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/wal
uuid: "ca8ca5ced019409bacfa9b2e62f58a58"
format_stamp: "Formatted at 2026-09-25 10:32:49 on dist-test-slave-dxdd"
server_key: "088e098e387c21ef4263362bc3b796f3"
server_key_iv: "aaf0b1bff2bdb61cbf1c70a9de583ceb"
server_key_version: "encryptionkey@0"
I20260925 10:32:50.277597 7007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:50.301723 7007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:50.305667 7007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:50.326187 7007 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:43321
I20260925 10:32:50.326234 7177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:43321 every 8 connection(s)
I20260925 10:32:50.327231 7007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-1/data/info.pb
I20260925 10:32:50.332208 7178 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:32:50.336404 7178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: Bootstrap starting.
I20260925 10:32:50.352874 7178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:32:50.353345 7178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: Bootstrap complete.
I20260925 10:32:50.354351 7178 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } }
I20260925 10:32:50.354605 7178 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca8ca5ced019409bacfa9b2e62f58a58, State: Initialized, Role: LEARNER
I20260925 10:32:50.354828 7178 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } }
I20260925 10:32:50.355647 7181 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } } }
I20260925 10:32:50.355890 7178 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:50.356010 7181 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:32:50.364078 7192 catalog_manager.cc:1259] Loaded cluster ID: 04db75e16edb4e8fa187793f7336f08a
I20260925 10:32:50.364169 7192 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: loading cluster ID for follower catalog manager: success
I20260925 10:32:50.367378 7192 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: acquiring CA information for follower catalog manager: success
I20260925 10:32:50.369526 7192 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 10:32:50.546108 7107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:50.546473 7107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:50.546614 7107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:50.557706 7107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:50.557866 7107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:50.557948 7107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:50.558032 7107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:50.558099 7107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:32:50.569214 7107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46413,127.0.97.61:43321,127.0.97.60:36757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:50.570376 7107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:50.571596 7107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:50.583578 7197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:50.583861 7200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:50.583925 7198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:50.584808 7107 server_base.cc:1034] running on GCE node
I20260925 10:32:50.585517 7107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:50.586146 7002 heartbeater.cc:503] Master 127.0.97.62:46413 was elected leader, sending a full tablet report...
I20260925 10:32:50.587044 7107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:50.588263 7107 hybrid_clock.cc:648] HybridClock initialized: now 1790332370588202 us; error 55 us; skew 500 ppm
I20260925 10:32:50.588658 7107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:50 dist-test-slave-dxdd krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332370, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:50.591603 7107 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260925 10:32:50.592890 7107 webserver.cc:460] Webserver started at http://127.0.97.60:34763/ using document root <none> and password file <none>
I20260925 10:32:50.593528 7107 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:50.593663 7107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:50.593937 7107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:50.596724 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/data/instance:
uuid: "e36414eb5d1d42b6a5461ee76d162ca6"
format_stamp: "Formatted at 2026-09-25 10:32:50 on dist-test-slave-dxdd"
server_key: "c153a0331a200be225a52dab2c0bf7eb"
server_key_iv: "c53d1fb80f3d966e248856676647f37c"
server_key_version: "encryptionkey@0"
I20260925 10:32:50.597440 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/wal/instance:
uuid: "e36414eb5d1d42b6a5461ee76d162ca6"
format_stamp: "Formatted at 2026-09-25 10:32:50 on dist-test-slave-dxdd"
server_key: "c153a0331a200be225a52dab2c0bf7eb"
server_key_iv: "c53d1fb80f3d966e248856676647f37c"
server_key_version: "encryptionkey@0"
I20260925 10:32:50.602216 7107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 10:32:50.605409 7207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:50.606709 7107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:32:50.606875 7107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/wal
uuid: "e36414eb5d1d42b6a5461ee76d162ca6"
format_stamp: "Formatted at 2026-09-25 10:32:50 on dist-test-slave-dxdd"
server_key: "c153a0331a200be225a52dab2c0bf7eb"
server_key_iv: "c53d1fb80f3d966e248856676647f37c"
server_key_version: "encryptionkey@0"
I20260925 10:32:50.607053 7107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:50.622076 7107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:50.625850 7107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:50.626128 7107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:50.645469 7107 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36757
I20260925 10:32:50.645470 7258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36757 every 8 connection(s)
I20260925 10:32:50.646950 7107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/data/info.pb
I20260925 10:32:50.649943 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 7107
I20260925 10:32:50.650471 7259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:50.660576 7259 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:32:50 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 25 10:32:50 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332370, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260925 10:32:50.674782 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.652014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.0.97.60:36757)
0925 10:32:50.652769 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.652783 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.652832 (+ 49us) server_negotiation.cc:407] Connection header received
0925 10:32:50.655045 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.655072 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.655584 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.656088 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.657809 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.659811 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.661021 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.662011 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.662193 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.667040 (+ 4847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.667049 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.667059 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.667093 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.670537 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.671201 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.671207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.671213 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.671348 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.671821 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.671825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.671827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.672357 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.672659 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.672994 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.673430 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":309,"threads_started":1}
I20260925 10:32:50.680367 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.663632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.0.97.62:46413)
0925 10:32:50.663818 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.663824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.664455 (+ 631us) server_negotiation.cc:407] Connection header received
0925 10:32:50.664661 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.664670 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.664731 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.664836 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.666019 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.666873 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.668306 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.668710 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.668838 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.672492 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.672499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.672503 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.672541 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.676238 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.677347 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.677353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.677356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.677507 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.678425 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.678431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.678436 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.678987 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.679148 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.679823 (+ 675us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.680020 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:32:50.680437 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.663504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:33192)
0925 10:32:50.664282 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:50.664339 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:50.664536 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:50.664889 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:50.664895 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:50.665521 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:50.665858 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.665871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.667048 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.667053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.668130 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.668141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.668902 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.668914 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.669077 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.669623 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:50.669643 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:50.672307 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:50.676382 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.676390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.676396 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.677170 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.678027 (+ 857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.678034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.678037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.678225 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.679267 (+ 1042us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:50.679275 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:50.679514 (+ 239us) client_negotiation.cc:769] Sending connection context
0925 10:32:50.679717 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 10:32:50.680009 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":462,"threads_started":1}
I20260925 10:32:50.682523 7259 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:32:50.684960 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.674390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43321 (local address 127.0.0.1:40206)
0925 10:32:50.678962 (+ 4572us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:50.678976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:50.679112 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:50.679690 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:50.679697 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:50.679723 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:50.679909 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.679916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.681351 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.681358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.683077 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.683093 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.684305 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.684309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.684417 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.684453 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 10:32:50.684526 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 10:32:50.684649 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4475,"thread_start_us":4370,"threads_started":1}
I20260925 10:32:50.685056 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.674511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.0.97.61:43321)
0925 10:32:50.678536 (+ 4025us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.678543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.679064 (+ 521us) server_negotiation.cc:407] Connection header received
0925 10:32:50.679240 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.679245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.679353 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.679471 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:50.680037 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.681233 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.683242 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.684175 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.684277 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.684404 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.684580 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.684729 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3894,"thread_start_us":3721,"threads_started":1}
I20260925 10:32:50.686048 7153 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260925 10:32:50.686702 7181 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8edbe7ca91c47a2ac5daed766a95286. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } } }
I20260925 10:32:50.687001 7181 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: This master's current role is: LEARNER
Sep 25 10:32:50 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332370, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260925 10:32:50.695878 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.683126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43321 (local address 127.0.0.1:40216)
0925 10:32:50.684544 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:50.684562 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:50.684656 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:50.684983 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:50.684988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:50.685676 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:50.686191 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.686199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.687544 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.687548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.688432 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.688441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.689082 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.689086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.689216 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.689686 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:50.689701 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:50.691496 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:50.694051 (+ 2555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.694054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.694057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.694452 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.694747 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.694750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.694752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.694863 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.695258 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:50.695261 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:50.695359 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 10:32:50.695456 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 10:32:50.695665 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1294}
I20260925 10:32:50.695933 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.683206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.0.97.61:43321)
0925 10:32:50.683632 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.683636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.684637 (+ 1001us) server_negotiation.cc:407] Connection header received
0925 10:32:50.684758 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.684763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.684837 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.684952 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.686378 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.687434 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.688569 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.688970 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.689060 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.691681 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.691685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.691688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.691716 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.693949 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.694539 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.694542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.694544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.694662 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.694942 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.694945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.694946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.695173 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.695279 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.695558 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.695695 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":199,"threads_started":1}
I20260925 10:32:50.696923 7259 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:32:50 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332370, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260925 10:32:50.709004 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.697518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36757 (local address 127.0.0.1:43146)
0925 10:32:50.697704 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:50.697718 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:50.697817 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:50.698107 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:50.698111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:50.698463 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:50.698578 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.698585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.699430 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.699437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.700208 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.700217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.700908 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.700913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.701096 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.701511 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:50.701525 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:50.703413 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:50.706563 (+ 3150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.706566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.706568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.706900 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.707258 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.707265 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.707271 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.707460 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.707918 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:50.707924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:50.708087 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 10:32:50.708251 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 10:32:50.708563 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":5}
I20260925 10:32:50.709038 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.697723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43146 (local address 127.0.97.60:36757)
0925 10:32:50.697863 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.697870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.697886 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:32:50.697923 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.697928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.697982 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.698066 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.698697 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.699321 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.700339 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.700733 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.700894 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.703552 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.703556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.703559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.703585 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.706437 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.707015 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.707022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.707025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.707155 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.707562 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.707566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.707568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.707805 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.707904 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.708323 (+ 419us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.708708 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":187,"server-negotiator.queue_time_us":52}
I20260925 10:32:50.713932 7259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: Bootstrap starting.
I20260925 10:32:50.717211 7259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:50.718175 7259 log.cc:824] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:50.720537 7259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: No bootstrap required, opened a new log
I20260925 10:32:50.727527 7259 raft_consensus.cc:348] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } }
I20260925 10:32:50.727873 7259 raft_consensus.cc:374] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:50.727995 7259 raft_consensus.cc:729] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e36414eb5d1d42b6a5461ee76d162ca6, State: Initialized, Role: FOLLOWER
I20260925 10:32:50.728502 7259 consensus_queue.cc:260] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } }
I20260925 10:32:50.729487 7268 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } } }
I20260925 10:32:50.729839 7268 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:32:50.730854 7259 sys_catalog.cc:564] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 10:32:50.739370 7279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:32:50.739524 7279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:32:50.743376 7107 master_runner.cc:428] Detected that this master e36414eb5d1d42b6a5461ee76d162ca6 is joining an existing cluster
I20260925 10:32:50.743507 7107 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:36757
I20260925 10:32:50.766060 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.747565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.0.97.61:43321)
0925 10:32:50.747730 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.747734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.748783 (+ 1049us) server_negotiation.cc:407] Connection header received
0925 10:32:50.749053 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.749058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.749151 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.749222 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.750798 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.751900 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.754836 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.755478 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.755583 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.759344 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.759373 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.759378 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.759417 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.763482 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.764155 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.764160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.764164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.764287 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.764667 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.764674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.764677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.764938 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.765087 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.765488 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.765688 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:32:50.766359 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.748012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43321 (local address 127.0.0.1:40232)
0925 10:32:50.748671 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:50.748696 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:50.748899 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:50.749430 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:50.749434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:50.750238 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:50.750622 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.750647 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.752073 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.752078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.754682 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.754695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.755854 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.755859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.756042 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.757166 (+ 1124us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:50.757193 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:50.759123 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:50.763614 (+ 4491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.763617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.763620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.764020 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.764397 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.764403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.764407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.764555 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.765060 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:50.765064 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:50.765255 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 10:32:50.765393 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 10:32:50.765851 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":116,"spinlock_wait_cycles":88832,"thread_start_us":384,"threads_started":1}
I20260925 10:32:50.769083 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.747396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.0.97.62:46413)
0925 10:32:50.747583 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.747591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.749146 (+ 1555us) server_negotiation.cc:407] Connection header received
0925 10:32:50.749360 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.749365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.749444 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.749507 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.750820 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.751900 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.754886 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.755582 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.755700 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.759238 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.759245 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.759249 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.759299 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.762769 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.766751 (+ 3982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.766757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.766760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.766912 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.767364 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.767369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.767372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.767732 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.767869 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.768465 (+ 596us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.768726 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:32:50.769126 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.748060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:33208)
0925 10:32:50.749056 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:50.749072 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:50.749229 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:50.749704 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:50.749713 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:50.750375 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:50.750678 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.750689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.752073 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.752078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.754705 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.754728 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.755966 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.755976 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.756332 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.757222 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:50.757249 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:50.758978 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:50.762942 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.762948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.762952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.765978 (+ 3026us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.767055 (+ 1077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.767062 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.767066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.767234 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.767925 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:50.767930 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:50.768146 (+ 216us) client_negotiation.cc:769] Sending connection context
0925 10:32:50.768330 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 10:32:50.768664 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":163,"spinlock_wait_cycles":269056,"thread_start_us":694,"threads_started":1}
I20260925 10:32:50.769493 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.749165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.0.97.60:36757)
0925 10:32:50.749324 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.749330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.754118 (+ 4788us) server_negotiation.cc:407] Connection header received
0925 10:32:50.754344 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.754348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.754424 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.754487 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.755731 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.756959 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.758951 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.759585 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.759691 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.762479 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.762487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.762492 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.762526 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.766591 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.767420 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.767425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.767428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.767563 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.767960 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.767968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.767970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.768273 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.768441 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.768919 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.769109 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:32:50.770206 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.749427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36757 (local address 127.0.0.1:43150)
0925 10:32:50.754040 (+ 4613us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:50.754057 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:50.754201 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:50.754646 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:50.754650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:50.755283 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:50.755593 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.755604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.757183 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.757187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.758811 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.758827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.759943 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.759949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.760111 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.760693 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:50.760745 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:50.761870 (+ 1125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:50.766839 (+ 4969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.766843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.766846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.767308 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.767716 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.767720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.767723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.767857 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.768504 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:50.768508 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:50.768683 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 10:32:50.769636 (+ 953us) client_negotiation.cc:240] Negotiation successful
0925 10:32:50.769886 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4498,"thread_start_us":4164,"threads_started":1}
Sep 25 10:32:50 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332367, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260925 10:32:50.781522 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 6607
I20260925 10:32:50.789134 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.774465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36757 (local address 127.0.0.1:43156)
0925 10:32:50.774610 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:50.774629 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:50.774723 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:50.775007 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:50.775015 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:50.775372 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:50.775505 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.775513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.776590 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.776595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.777690 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.777701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.778675 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.778683 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.778864 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.779447 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:50.779472 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:50.782517 (+ 3045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:50.786797 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.786802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.786806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.787233 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.787583 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.787589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.787592 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.787725 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.788203 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:50.788208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:50.788358 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 10:32:50.788497 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 10:32:50.788774 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 10:32:50.789139 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.774517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.0.97.60:36757)
0925 10:32:50.774659 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.774662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.774676 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:32:50.774819 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.774822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.774878 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.774987 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.775635 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.776461 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.777893 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.778515 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.778648 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.782755 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.782762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.782769 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.782804 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.786649 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.787356 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.787359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.787361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.787465 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.787828 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.787835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.787838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.788085 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.788239 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.788578 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.788783 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:32:50.791239 6546 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:36757: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 36757
}
attrs {
promote: true
}
}
dest_uuid: "e8edbe7ca91c47a2ac5daed766a95286"
cas_config_opid_index: 5
I20260925 10:32:50.792382 6546 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } }
Sep 25 10:32:50 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260925 10:32:50.795084 7103 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e36414eb5d1d42b6a5461ee76d162ca6 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } } }
I20260925 10:32:50.799057 7153 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 LEARNER]: Refusing update from remote peer e8edbe7ca91c47a2ac5daed766a95286: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 10:32:50.799933 7234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:50.803571 7234 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 FOLLOWER]: Refusing update from remote peer e8edbe7ca91c47a2ac5daed766a95286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 10:32:50.808926 7290 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 10:32:50.810700 7289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 10:32:50.814774 7103 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:36757
I20260925 10:32:50.815521 7107 master.cc:537] Master@127.0.97.60:36757 shutting down...
I20260925 10:32:50.818172 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.782412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.0.97.1:33123)
0925 10:32:50.783024 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.783036 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.783066 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:32:50.783136 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.783157 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.783449 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.783615 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.785153 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.786851 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.788722 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.789618 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.789794 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.808241 (+ 18447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.808262 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.808282 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.808337 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.814222 (+ 5885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.815160 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.815168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.815171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.815311 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.815783 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.815788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.815791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.816333 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.816714 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.817540 (+ 826us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.817812 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":520,"threads_started":1}
I20260925 10:32:50.819897 6730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:50.823601 7153 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e36414eb5d1d42b6a5461ee76d162ca6 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } } }
I20260925 10:32:50.824471 7291 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } } }
I20260925 10:32:50.824999 7291 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:50.829380 7181 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } } }
I20260925 10:32:50.829639 7181 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:32:50.830605 7103 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286: attempting to promote NON_VOTER ca8ca5ced019409bacfa9b2e62f58a58 to VOTER
I20260925 10:32:50.831386 7103 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } }
I20260925 10:32:50.835521 7234 raft_consensus.cc:858] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } }
W20260925 10:32:50.835716 7234 raft_consensus.cc:1565] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } }
I20260925 10:32:50.836162 7153 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 LEARNER]: Refusing update from remote peer e8edbe7ca91c47a2ac5daed766a95286: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 10:32:50.836750 7290 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 10:32:50.839344 7290 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ca8ca5ced019409bacfa9b2e62f58a58 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } } }
I20260925 10:32:50.843111 7153 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ca8ca5ced019409bacfa9b2e62f58a58 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } } }
I20260925 10:32:50.844832 7103 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } } }
I20260925 10:32:50.845130 7103 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:50.845408 7290 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } } }
I20260925 10:32:50.846136 7290 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:50.847368 7181 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } } }
I20260925 10:32:50.847623 7181 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:32:50.854380 6531 consensus_peers.cc:487] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 -> Peer e36414eb5d1d42b6a5461ee76d162ca6 (127.0.97.60:36757): Couldn't send request to peer e36414eb5d1d42b6a5461ee76d162ca6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 10:32:50.857483 7234 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ca8ca5ced019409bacfa9b2e62f58a58 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } } }
I20260925 10:32:50.874141 7302 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } } }
I20260925 10:32:50.875034 7302 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 10:32:50.875658 7268 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8edbe7ca91c47a2ac5daed766a95286. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } } }
I20260925 10:32:50.875608 7298 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: true } } }
I20260925 10:32:50.875891 7268 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 10:32:50.875916 7298 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 10:32:50.878474 7107 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260925 10:32:50.878927 7107 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260925 10:32:50.879102 7107 tablet_replica.cc:330] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: stopping tablet replica
I20260925 10:32:50.887315 7107 master.cc:552] Master@127.0.97.60:36757 shutdown complete.
I20260925 10:32:50.887399 7107 master_runner.cc:305] Clearing existing system tablet
I20260925 10:32:50.887842 7107 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:50.887955 7107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:50.893687 7107 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 10:32:50.895361 7304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:50.895809 7107 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:50.896093 7107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:32:50.896237 7107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/wal
uuid: "e36414eb5d1d42b6a5461ee76d162ca6"
format_stamp: "Formatted at 2026-09-25 10:32:50 on dist-test-slave-dxdd"
server_key: "c153a0331a200be225a52dab2c0bf7eb"
server_key_iv: "c53d1fb80f3d966e248856676647f37c"
server_key_version: "encryptionkey@0"
I20260925 10:32:50.900768 7107 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:50.907418 7107 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 10:32:50.907574 7107 log.cc:1196] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 10:32:50.908033 7107 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: Deleting consensus metadata
I20260925 10:32:50.908289 7107 master_runner.cc:315] Copying system tablet from 127.0.97.62:46413
I20260925 10:32:50.911296 7107 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46413
I20260925 10:32:50.928434 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.912099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33216 (local address 127.0.97.62:46413)
0925 10:32:50.912323 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:50.912330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:50.914880 (+ 2550us) server_negotiation.cc:407] Connection header received
0925 10:32:50.915053 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:50.915059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:50.915137 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:50.915239 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:50.916236 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.917315 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.919267 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.919862 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.919985 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.922270 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:50.922276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:50.922281 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:50.922350 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:50.925634 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.926406 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.926412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.926416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.926558 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:50.926970 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:50.926975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:50.926978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:50.927284 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:50.927440 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:50.927889 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 10:32:50.928098 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 10:32:50.929121 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:50.912064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:33216)
0925 10:32:50.914748 (+ 2684us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:50.914768 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:50.914909 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:50.915286 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:50.915291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:50.915805 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:50.916066 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.916077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.917454 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.917459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.919100 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:50.919115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:50.920020 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:50.920026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:50.920213 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:50.920957 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:50.920980 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:50.922063 (+ 1083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:50.925766 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.925773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.925778 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.926258 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.926679 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:50.926685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:50.926694 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:50.926858 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:50.927414 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:50.927419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:50.927626 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 10:32:50.927772 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 10:32:50.928097 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2529,"thread_start_us":2427,"threads_started":1}
I20260925 10:32:50.929667 6566 tablet_copy_service.cc:140] P e8edbe7ca91c47a2ac5daed766a95286: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e36414eb5d1d42b6a5461ee76d162ca6 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:33216)
I20260925 10:32:50.929831 6566 tablet_copy_service.cc:161] P e8edbe7ca91c47a2ac5daed766a95286: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e36414eb5d1d42b6a5461ee76d162ca6 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:33216: session id = e36414eb5d1d42b6a5461ee76d162ca6-00000000000000000000000000000000
I20260925 10:32:50.933240 6566 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 10:32:50.935014 7107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 10:32:50.941996 7107 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: tablet copy: Starting download of 0 data blocks...
I20260925 10:32:50.942268 7107 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: tablet copy: Starting download of 1 WAL segments...
I20260925 10:32:50.944087 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 6607
I20260925 10:32:50.944826 7107 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 10:32:50.948890 6566 tablet_copy_service.cc:342] P e8edbe7ca91c47a2ac5daed766a95286: Request end of tablet copy session e36414eb5d1d42b6a5461ee76d162ca6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:33216
I20260925 10:32:50.949046 6566 tablet_copy_service.cc:433] P e8edbe7ca91c47a2ac5daed766a95286: ending tablet copy session e36414eb5d1d42b6a5461ee76d162ca6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e36414eb5d1d42b6a5461ee76d162ca6
I20260925 10:32:50.966815 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33123
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46413
--webserver_port=32777
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46413,127.0.97.61:43321
--tserver_master_addrs=127.0.97.62:46413,127.0.97.61:43321,127.0.97.60:36757 with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:32:50.969069 7107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:50.974840 7311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:50.999181 7313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:51.014547 7315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:51.018728 7107 server_base.cc:1034] running on GCE node
I20260925 10:32:51.019385 7107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:51.027184 7107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:51.028471 7107 hybrid_clock.cc:648] HybridClock initialized: now 1790332371028411 us; error 53 us; skew 500 ppm
I20260925 10:32:51.028858 7107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:51 dist-test-slave-dxdd krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332371, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:51.031332 7107 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260925 10:32:51.052616 7107 webserver.cc:460] Webserver started at http://127.0.97.60:33819/ using document root <none> and password file <none>
I20260925 10:32:51.053164 7107 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:51.053350 7107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:51.066440 7107 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.004s sys 0.004s
I20260925 10:32:51.077893 7322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:51.085352 7107 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.006s sys 0.004s
I20260925 10:32:51.085608 7107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/wal
uuid: "e36414eb5d1d42b6a5461ee76d162ca6"
format_stamp: "Formatted at 2026-09-25 10:32:50 on dist-test-slave-dxdd"
server_key: "c153a0331a200be225a52dab2c0bf7eb"
server_key_iv: "c53d1fb80f3d966e248856676647f37c"
server_key_version: "encryptionkey@0"
I20260925 10:32:51.085920 7107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:51.163041 7107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:51.168222 7107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:51.383167 7107 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36757
I20260925 10:32:51.383220 7374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36757 every 8 connection(s)
I20260925 10:32:51.384601 7107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/master-2/data/info.pb
I20260925 10:32:51.389914 7375 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:32:51.393513 7375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: Bootstrap starting.
I20260925 10:32:51.417863 7375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:32:51.418738 7375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: Bootstrap complete.
I20260925 10:32:51.419622 7375 raft_consensus.cc:348] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } }
I20260925 10:32:51.419850 7375 raft_consensus.cc:729] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e36414eb5d1d42b6a5461ee76d162ca6, State: Initialized, Role: LEARNER
I20260925 10:32:51.420173 7375 consensus_queue.cc:260] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } }
I20260925 10:32:51.421128 7380 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } } }
I20260925 10:32:51.421449 7375 sys_catalog.cc:564] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:51.421550 7380 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:32:51.422961 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:51.402082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43164 (local address 127.0.97.60:36757)
0925 10:32:51.407217 (+ 5135us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:51.407223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:51.407715 (+ 492us) server_negotiation.cc:407] Connection header received
0925 10:32:51.407967 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:51.407972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:51.408066 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:51.408203 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:51.409192 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.410818 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.412851 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.413482 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.413606 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:51.415803 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:51.415810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:51.415815 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:51.415851 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:51.419850 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:51.420700 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:51.420706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:51.420710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:51.420881 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:51.421362 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:51.421366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:51.421370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:51.421723 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:51.421871 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:51.422340 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 10:32:51.422559 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4976,"thread_start_us":4819,"threads_started":1}
I20260925 10:32:51.423112 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:51.397607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36757 (local address 127.0.0.1:43164)
0925 10:32:51.407554 (+ 9947us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:51.407574 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:51.407810 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:51.408265 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:51.408271 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:51.408755 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:51.409002 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:51.409016 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.410959 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.410967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:51.412684 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:51.412699 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.413652 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.413657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:51.413800 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:51.414570 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:51.414595 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:51.415576 (+ 981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:51.420008 (+ 4432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:51.420013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:51.420017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:51.420558 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:51.421036 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:51.421041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:51.421045 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:51.421214 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:51.421862 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:51.421867 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:51.422030 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 10:32:51.422210 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 10:32:51.422547 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9824,"thread_start_us":9733,"threads_started":1}
I20260925 10:32:51.424151 7349 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 10:32:51.424793 7380 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8edbe7ca91c47a2ac5daed766a95286. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } } }
I20260925 10:32:51.425451 7380 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:32:51.430364 7391 catalog_manager.cc:1259] Loaded cluster ID: 04db75e16edb4e8fa187793f7336f08a
I20260925 10:32:51.430441 7391 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: loading cluster ID for follower catalog manager: success
I20260925 10:32:51.432668 7391 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: acquiring CA information for follower catalog manager: success
I20260925 10:32:51.434525 7391 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 10:32:51.440763 7312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:51.441030 7312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:51.441200 7312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:51.448288 7312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:51.448426 7312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:51.448594 7312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:32:51.458693 7312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=32777
--tserver_master_addrs=127.0.97.62:46413,127.0.97.61:43321,127.0.97.60:36757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:51.460045 7312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:51.461323 7312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:51.474058 7397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:51.474099 7399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:51.474135 7396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:51.474664 7312 server_base.cc:1034] running on GCE node
I20260925 10:32:51.475296 7312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:51.476132 7312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:51.477421 7312 hybrid_clock.cc:648] HybridClock initialized: now 1790332371477354 us; error 65 us; skew 500 ppm
I20260925 10:32:51.477833 7312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:51 dist-test-slave-dxdd krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332371, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:51.481278 7312 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260925 10:32:51.482722 7312 webserver.cc:460] Webserver started at http://127.0.97.1:32777/ using document root <none> and password file <none>
I20260925 10:32:51.483371 7312 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:51.483523 7312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:51.488658 7312 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 10:32:51.491709 7406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:51.492861 7312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:32:51.493029 7312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "65690ad1c05d4a8083522828aca777ba"
format_stamp: "Formatted at 2026-09-25 10:32:48 on dist-test-slave-dxdd"
server_key: "fa751b10999b0e6574e78669ab40a4a8"
server_key_iv: "4b3747e894739cc4787931bce6df995a"
server_key_version: "encryptionkey@0"
I20260925 10:32:51.493961 7312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:51.535903 7312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:51.539676 7312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:51.539885 7312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:51.540712 7312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:51.542373 7312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:51.542480 7312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 10:32:51.542553 7312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:51.542603 7312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:51.576982 7312 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33123
I20260925 10:32:51.577073 7518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33123 every 8 connection(s)
I20260925 10:32:51.578614 7312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:32:51.582376 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 7312
I20260925 10:32:51.582700 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 6741
Sep 25 10:32:51 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Sep 25 10:32:51 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332371, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 25 10:32:51 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332371, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260925 10:32:51.645174 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:51.584035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36124 (local address 127.0.97.2:33031)
0925 10:32:51.620650 (+ 36615us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:51.620663 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:51.620701 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:32:51.620786 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:51.620797 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:51.621168 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:51.621368 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:51.624938 (+ 3570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.627283 (+ 2345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.629252 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.629783 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.629931 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:51.636656 (+ 6725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:51.636669 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:51.636685 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:51.636724 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:51.641085 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:51.642003 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:51.642014 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:51.642019 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:51.642186 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:51.642658 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:51.642663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:51.642665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:51.643647 (+ 982us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:51.643866 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:51.644449 (+ 583us) server_negotiation.cc:299] Negotiation successful
0925 10:32:51.644719 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36380,"thread_start_us":37790,"threads_started":1}
Sep 25 10:32:51 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332371, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260925 10:32:51.647296 6864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:51.654794 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:51.623541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56565 (local address 127.0.97.60:36757)
0925 10:32:51.623781 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:51.623790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:51.627032 (+ 3242us) server_negotiation.cc:407] Connection header received
0925 10:32:51.628410 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:51.628415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:51.628492 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:51.628556 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:51.632508 (+ 3952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.636395 (+ 3887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.638328 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.638962 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.639081 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:51.644898 (+ 5817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:51.644905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:51.644910 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:51.644953 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:51.648923 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:51.650731 (+ 1808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:51.650738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:51.650743 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:51.650875 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:51.651409 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:51.651415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:51.651418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:51.652259 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:51.652408 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:51.654189 (+ 1781us) server_negotiation.cc:299] Negotiation successful
0925 10:32:51.654439 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 10:32:51.656319 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:51.623377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36757 (local address 127.0.97.1:56565)
0925 10:32:51.626911 (+ 3534us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:51.626931 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:51.627667 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:51.628752 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:51.628756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:51.630505 (+ 1749us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:51.631850 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:51.631864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.636531 (+ 4667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.636536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:51.638110 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:51.638128 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.639285 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.639291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:51.639447 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:51.641507 (+ 2060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:51.641541 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:51.644320 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:51.649226 (+ 4906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:51.649259 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:51.649281 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:51.650585 (+ 1304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:51.651026 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:51.651030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:51.651033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:51.651293 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:51.653061 (+ 1768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:51.653071 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:51.653863 (+ 792us) client_negotiation.cc:769] Sending connection context
0925 10:32:51.655279 (+ 1416us) client_negotiation.cc:240] Negotiation successful
0925 10:32:51.655587 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3402,"mutex_wait_us":601,"spinlock_wait_cycles":20992,"thread_start_us":3709,"threads_started":1}
I20260925 10:32:51.656656 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:51.581471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54671 (local address 127.0.97.61:43321)
0925 10:32:51.627192 (+ 45721us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:51.627198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:51.627224 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:32:51.628096 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:51.628100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:51.628219 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:51.628321 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:51.632013 (+ 3692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.633140 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.635999 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.636692 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.636811 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:51.646542 (+ 9731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:51.646548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:51.646553 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:51.646591 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:51.651497 (+ 4906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:51.652319 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:51.652325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:51.652329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:51.652470 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:51.652888 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:51.652893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:51.652897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:51.653261 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:51.653401 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:51.655956 (+ 2555us) server_negotiation.cc:299] Negotiation successful
0925 10:32:51.656234 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45603,"thread_start_us":45957,"threads_started":1}
I20260925 10:32:51.657322 7526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:51.583276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43321 (local address 127.0.97.1:54671)
0925 10:32:51.626000 (+ 42724us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:51.626023 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:51.627660 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:51.628487 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:51.628490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:51.630502 (+ 2012us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:51.631733 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:51.631768 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.633367 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.633374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:51.635847 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:51.635861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.637087 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.637095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:51.637428 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:51.642997 (+ 5569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:51.643012 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:51.645161 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:51.651667 (+ 6506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:51.651672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:51.651677 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:51.652143 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:51.652564 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:51.652569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:51.652572 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:51.652749 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:51.653462 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:51.653468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:51.653681 (+ 213us) client_negotiation.cc:769] Sending connection context
0925 10:32:51.654011 (+ 330us) client_negotiation.cc:240] Negotiation successful
0925 10:32:51.654603 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42596,"mutex_wait_us":660,"thread_start_us":42432,"threads_started":1}
I20260925 10:32:51.658088 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:51.582688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.1:32951)
0925 10:32:51.622633 (+ 39945us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:51.622693 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:51.627660 (+ 4967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:51.628455 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:51.628468 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:51.630502 (+ 2034us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:51.631927 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:51.631937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.639721 (+ 7784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.639727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:51.641230 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:51.641243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.644060 (+ 2817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.644066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:51.644251 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:51.645421 (+ 1170us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:51.645436 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:51.651079 (+ 5643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:51.655274 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:51.655279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:51.655282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:51.655725 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:51.656133 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:51.656138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:51.656141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:51.656318 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:51.657008 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:51.657012 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:51.657263 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 10:32:51.657412 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 10:32:51.657686 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39641,"mutex_wait_us":402,"spinlock_wait_cycles":3968,"thread_start_us":39401,"threads_started":1}
I20260925 10:32:51.658547 7519 heartbeater.cc:347] Connected to a master server at 127.0.97.60:36757
I20260925 10:32:51.658872 7519 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:51.658898 7521 heartbeater.cc:347] Connected to a master server at 127.0.97.61:43321
I20260925 10:32:51.659039 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:51.581366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32951 (local address 127.0.97.62:46413)
0925 10:32:51.627571 (+ 46205us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:51.627578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:51.627597 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:32:51.627981 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:51.627987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:51.628065 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:51.628173 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:51.637043 (+ 8870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.639436 (+ 2393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.641953 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:51.642586 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:51.642722 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:51.651308 (+ 8586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:51.651316 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:51.651323 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:51.651381 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:51.655101 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:51.655859 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:51.655865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:51.655868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:51.656009 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:51.656459 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:51.656466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:51.656471 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:51.656792 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:51.656921 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:51.658453 (+ 1532us) server_negotiation.cc:299] Negotiation successful
0925 10:32:51.658690 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46078,"thread_start_us":45969,"threads_started":1}
I20260925 10:32:51.659047 7521 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:51.659771 7520 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46413
I20260925 10:32:51.659816 7519 heartbeater.cc:511] Master 127.0.97.60:36757 requested a full tablet report, sending...
I20260925 10:32:51.659906 7520 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:51.660266 7520 heartbeater.cc:511] Master 127.0.97.62:46413 requested a full tablet report, sending...
I20260925 10:32:51.660813 7521 heartbeater.cc:511] Master 127.0.97.61:43321 requested a full tablet report, sending...
I20260925 10:32:51.661613 6546 ts_manager.cc:194] Re-registered known tserver with Master: 65690ad1c05d4a8083522828aca777ba (127.0.97.1:33123)
I20260925 10:32:51.662245 7339 ts_manager.cc:194] Registered new tserver with Master: 65690ad1c05d4a8083522828aca777ba (127.0.97.1:33123)
I20260925 10:32:51.662556 7143 ts_manager.cc:194] Registered new tserver with Master: 65690ad1c05d4a8083522828aca777ba (127.0.97.1:33123)
I20260925 10:32:51.662698 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:32951
I20260925 10:32:51.738102 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 6741
I20260925 10:32:51.753798 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33031
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46413
--webserver_port=34933
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46413,127.0.97.61:43321
--tserver_master_addrs=127.0.97.62:46413,127.0.97.61:43321,127.0.97.60:36757 with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:32:51.975479 7370 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: true } } }
I20260925 10:32:51.975865 7370 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:51.977428 7532 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286: attempting to promote NON_VOTER e36414eb5d1d42b6a5461ee76d162ca6 to VOTER
I20260925 10:32:51.978053 7532 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } }
I20260925 10:32:51.980489 7153 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 FOLLOWER]: Refusing update from remote peer e8edbe7ca91c47a2ac5daed766a95286: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 10:32:51.980551 7349 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 LEARNER]: Refusing update from remote peer e8edbe7ca91c47a2ac5daed766a95286: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 10:32:51.981112 7532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 10:32:51.981508 7533 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 10:32:51.984608 7370 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e36414eb5d1d42b6a5461ee76d162ca6 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } } }
I20260925 10:32:51.985450 7349 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e36414eb5d1d42b6a5461ee76d162ca6 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } } }
I20260925 10:32:51.985606 7153 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e36414eb5d1d42b6a5461ee76d162ca6 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } } }
I20260925 10:32:51.988797 7533 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } } }
I20260925 10:32:51.989053 7533 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:51.989372 7370 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } } }
I20260925 10:32:51.989662 7370 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:51.989909 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } } }
I20260925 10:32:51.990151 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:51.994851 7539 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } } }
I20260925 10:32:51.995308 7539 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:32:51.995817 7538 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8edbe7ca91c47a2ac5daed766a95286" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } } }
I20260925 10:32:51.996117 7538 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:32:52.080008 7531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:52.080399 7531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:52.080556 7531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:52.087906 7531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:52.088017 7531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:52.088163 7531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:32:52.097388 7531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34933
--tserver_master_addrs=127.0.97.62:46413,127.0.97.61:43321,127.0.97.60:36757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:52.098758 7531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:52.100018 7531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:52.112776 7544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:52.112797 7545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:52.113415 7547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:52.113673 7531 server_base.cc:1034] running on GCE node
I20260925 10:32:52.114248 7531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:52.115165 7531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:52.116411 7531 hybrid_clock.cc:648] HybridClock initialized: now 1790332372116363 us; error 60 us; skew 500 ppm
I20260925 10:32:52.116765 7531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:52 dist-test-slave-dxdd krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:52.124866 7531 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260925 10:32:52.126484 7531 webserver.cc:460] Webserver started at http://127.0.97.2:34933/ using document root <none> and password file <none>
I20260925 10:32:52.127183 7531 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:52.127337 7531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:52.132524 7531 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 10:32:52.135277 7554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:52.136468 7531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:32:52.136657 7531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "2ba74a37e8f444618ecb68f3c6a44c88"
format_stamp: "Formatted at 2026-09-25 10:32:49 on dist-test-slave-dxdd"
server_key: "6677dfdb0d5ce0742f615636234e9742"
server_key_iv: "b9dffdb262f8cda32da9108c0cb281af"
server_key_version: "encryptionkey@0"
I20260925 10:32:52.137598 7531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:52.166190 7531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:52.170204 7531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:52.170509 7531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:52.171466 7531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:52.172840 7531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:52.172959 7531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:52.173087 7531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:52.173172 7531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:52.206110 7531 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33031
I20260925 10:32:52.206163 7666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33031 every 8 connection(s)
I20260925 10:32:52.207538 7531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:32:52.214812 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 7531
I20260925 10:32:52.215163 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 6874
Sep 25 10:32:52 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 25 10:32:52 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 25 10:32:52 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 25 10:32:52 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260925 10:32:52.256798 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.223449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52745 (local address 127.0.97.62:46413)
0925 10:32:52.226813 (+ 3364us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:52.226822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:52.235028 (+ 8206us) server_negotiation.cc:407] Connection header received
0925 10:32:52.236789 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:52.236797 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:52.236900 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:52.237011 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:52.239682 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.240449 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.243244 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.243642 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.243773 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.250110 (+ 6337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:52.250116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:52.250122 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:52.250169 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:52.253588 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.254398 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.254405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.254410 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.254550 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.255006 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.255014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.255020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.255364 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:52.255496 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:52.256075 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 10:32:52.256289 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3209,"thread_start_us":3115,"threads_started":1}
I20260925 10:32:52.257903 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.211789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43321 (local address 127.0.97.2:59113)
0925 10:32:52.233297 (+ 21508us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:52.233382 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:52.236262 (+ 2880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:52.237638 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:52.237643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:52.238439 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:52.239490 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.239513 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.241040 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.241046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.242888 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.242902 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.243934 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.243944 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.244166 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.245000 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:52.245023 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:52.248570 (+ 3547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:52.252438 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:52.252452 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:52.252468 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:52.253047 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:52.253481 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:52.253486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:52.253489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:52.253695 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:52.254386 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:52.254394 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:52.254790 (+ 396us) client_negotiation.cc:769] Sending connection context
0925 10:32:52.255106 (+ 316us) client_negotiation.cc:240] Negotiation successful
0925 10:32:52.255645 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21272,"mutex_wait_us":34,"spinlock_wait_cycles":1792,"thread_start_us":22525,"threads_started":1}
I20260925 10:32:52.257959 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.210345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57303 (local address 127.0.97.60:36757)
0925 10:32:52.237929 (+ 27584us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:52.237935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:52.237952 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:32:52.238016 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:52.238020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:52.238093 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:52.238879 (+ 786us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:52.240790 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.241883 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.244548 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.245174 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.245294 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.251527 (+ 6233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:52.251534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:52.251539 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:52.251577 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:52.254906 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.255700 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.255704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.255707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.255848 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.256277 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.256281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.256284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.256593 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:52.257252 (+ 659us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:52.257377 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 10:32:52.257574 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27453,"thread_start_us":26576,"threads_started":1}
I20260925 10:32:52.258348 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.210357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59113 (local address 127.0.97.61:43321)
0925 10:32:52.237293 (+ 26936us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:52.237300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:52.237320 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:32:52.237369 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:52.237374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:52.237458 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:52.237574 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:52.239702 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.240847 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.243054 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.243720 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.244542 (+ 822us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.248770 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:52.248782 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:52.248787 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:52.248822 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:52.252285 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.253181 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.253186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.253189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.253349 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.253825 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.253831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.253834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.254216 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:52.256847 (+ 2631us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:52.256989 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 10:32:52.257190 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26831,"thread_start_us":26741,"threads_started":1}
I20260925 10:32:52.258577 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.211111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36757 (local address 127.0.97.2:57303)
0925 10:32:52.233297 (+ 22186us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:52.233382 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:52.236768 (+ 3386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:52.238275 (+ 1507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:52.238281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:52.238757 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:52.239526 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.239540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.242024 (+ 2484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.242029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.244341 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.244354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.245588 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.245595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.245799 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.246364 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:52.246390 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:52.251322 (+ 4932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:52.255050 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:52.255057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:52.255061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:52.255562 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:52.255975 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:52.255980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:52.255983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:52.256137 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:52.256770 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:52.256776 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:52.256937 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 10:32:52.257054 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 10:32:52.257619 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21813,"mutex_wait_us":380,"spinlock_wait_cycles":47360,"thread_start_us":10329,"threads_started":1}
I20260925 10:32:52.258889 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.216159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.0.97.3:37187)
0925 10:32:52.234897 (+ 18738us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:52.234910 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:52.234949 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:32:52.235038 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:52.235049 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:52.235372 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:52.235580 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:52.238072 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.239651 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.241433 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.242209 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.242340 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.249936 (+ 7596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:52.249948 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:52.249964 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:52.250008 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:52.254287 (+ 4279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.255148 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.255156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.255160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.255306 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.255731 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.255736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.255738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.256116 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:52.256375 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:52.257758 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0925 10:32:52.258512 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18523,"thread_start_us":18374,"threads_started":1}
I20260925 10:32:52.259073 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.225964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.2:52745)
0925 10:32:52.234931 (+ 8967us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:52.234948 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:52.236678 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:52.237266 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:52.237277 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:52.238438 (+ 1161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:52.239527 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.239540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.240594 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.240601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.243137 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.243146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.243866 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.243873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.244153 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.244932 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:52.244967 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:52.249955 (+ 4988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:52.253749 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:52.253754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:52.253759 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:52.254221 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:52.254697 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:52.254703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:52.254709 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:52.254860 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:52.255567 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:52.255572 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:52.255791 (+ 219us) client_negotiation.cc:769] Sending connection context
0925 10:32:52.257074 (+ 1283us) client_negotiation.cc:240] Negotiation successful
0925 10:32:52.257323 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8822,"mutex_wait_us":110,"spinlock_wait_cycles":4480,"thread_start_us":8396,"threads_started":1}
I20260925 10:32:52.260264 6997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:52.260397 7668 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46413
I20260925 10:32:52.260605 7669 heartbeater.cc:347] Connected to a master server at 127.0.97.61:43321
I20260925 10:32:52.260712 7668 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:52.260742 7669 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:52.261294 7667 heartbeater.cc:347] Connected to a master server at 127.0.97.60:36757
I20260925 10:32:52.261441 7667 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:52.261677 7669 heartbeater.cc:511] Master 127.0.97.61:43321 requested a full tablet report, sending...
I20260925 10:32:52.261703 7668 heartbeater.cc:511] Master 127.0.97.62:46413 requested a full tablet report, sending...
I20260925 10:32:52.261910 7667 heartbeater.cc:511] Master 127.0.97.60:36757 requested a full tablet report, sending...
I20260925 10:32:52.263408 7339 ts_manager.cc:194] Registered new tserver with Master: 2ba74a37e8f444618ecb68f3c6a44c88 (127.0.97.2:33031)
I20260925 10:32:52.263599 7143 ts_manager.cc:194] Registered new tserver with Master: 2ba74a37e8f444618ecb68f3c6a44c88 (127.0.97.2:33031)
I20260925 10:32:52.264518 6546 ts_manager.cc:194] Re-registered known tserver with Master: 2ba74a37e8f444618ecb68f3c6a44c88 (127.0.97.2:33031)
I20260925 10:32:52.265868 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52745
I20260925 10:32:52.363688 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 6874
I20260925 10:32:52.379784 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37187
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46413
--webserver_port=37817
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46413,127.0.97.61:43321
--tserver_master_addrs=127.0.97.62:46413,127.0.97.61:43321,127.0.97.60:36757 with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:32:52.665060 7520 heartbeater.cc:503] Master 127.0.97.62:46413 was elected leader, sending a full tablet report...
W20260925 10:32:52.699009 7680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:52.699250 7680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:52.699340 7680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:52.705999 7680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:52.706096 7680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:52.706200 7680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:32:52.715417 7680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37817
--tserver_master_addrs=127.0.97.62:46413,127.0.97.61:43321,127.0.97.60:36757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:52.716621 7680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:52.717833 7680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:52.731410 7685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:52.731462 7688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:52.731387 7686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:52.732290 7680 server_base.cc:1034] running on GCE node
I20260925 10:32:52.732829 7680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:52.733662 7680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:52.734875 7680 hybrid_clock.cc:648] HybridClock initialized: now 1790332372734838 us; error 50 us; skew 500 ppm
I20260925 10:32:52.735168 7680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:52 dist-test-slave-dxdd krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:52.737984 7680 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260925 10:32:52.739307 7680 webserver.cc:460] Webserver started at http://127.0.97.3:37817/ using document root <none> and password file <none>
I20260925 10:32:52.739951 7680 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:52.740096 7680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:52.745096 7680 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 10:32:52.747498 7695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:52.748622 7680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:32:52.748791 7680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "9ba24b6ac5a349b3b3a88387a76dba5c"
format_stamp: "Formatted at 2026-09-25 10:32:49 on dist-test-slave-dxdd"
server_key: "e1ac8a5f7e3dbb6ba0b4be67bd00af8f"
server_key_iv: "07ed818589754b3acf6ab66b5ddccfc5"
server_key_version: "encryptionkey@0"
I20260925 10:32:52.749660 7680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:52.762014 7680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:52.766055 7680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:52.766379 7680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:52.767657 7680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:32:52.768870 7680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:52.768951 7680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:52.769022 7680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:52.769075 7680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:52.804718 7680 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37187
I20260925 10:32:52.804801 7807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37187 every 8 connection(s)
I20260925 10:32:52.806128 7680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:32:52.812340 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 7680
Sep 25 10:32:52 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 25 10:32:52 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260925 10:32:52.861850 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.823724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55767 (local address 127.0.97.60:36757)
0925 10:32:52.837809 (+ 14085us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:52.837818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:52.838658 (+ 840us) server_negotiation.cc:407] Connection header received
0925 10:32:52.838910 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:52.838915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:52.838995 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:52.839071 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:52.841888 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.843201 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.846451 (+ 3250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.847308 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.848373 (+ 1065us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.852529 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:52.852538 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:52.852543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:52.852586 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:52.856639 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.857541 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.857548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.857552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.857706 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.858293 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.858298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.858335 (+ 37us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.858919 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:52.859051 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:52.860855 (+ 1804us) server_negotiation.cc:299] Negotiation successful
0925 10:32:52.861509 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13913,"thread_start_us":14097,"threads_started":1}
I20260925 10:32:52.862653 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.829766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36757 (local address 127.0.97.3:55767)
0925 10:32:52.838536 (+ 8770us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:52.838580 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:52.838753 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:52.840040 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:52.840051 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:52.840800 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:52.841699 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.841725 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.843431 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.843440 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.846154 (+ 2714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.846174 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.847603 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.847614 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.847953 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.848710 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:52.848740 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:52.852300 (+ 3560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:52.856799 (+ 4499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:52.856808 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:52.856827 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:52.857403 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:52.857835 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:52.857844 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:52.857847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:52.858147 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:52.859152 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:52.859163 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:52.859602 (+ 439us) client_negotiation.cc:769] Sending connection context
0925 10:32:52.859917 (+ 315us) client_negotiation.cc:240] Negotiation successful
0925 10:32:52.860463 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8651,"mutex_wait_us":2,"thread_start_us":6865,"threads_started":1}
I20260925 10:32:52.864867 7808 heartbeater.cc:347] Connected to a master server at 127.0.97.60:36757
I20260925 10:32:52.865216 7808 heartbeater.cc:464] Registering TS with master...
Sep 25 10:32:52 dist-test-slave-dxdd krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260925 10:32:52.866140 7808 heartbeater.cc:511] Master 127.0.97.60:36757 requested a full tablet report, sending...
I20260925 10:32:52.868088 7339 ts_manager.cc:194] Registered new tserver with Master: 9ba24b6ac5a349b3b3a88387a76dba5c (127.0.97.3:37187)
I20260925 10:32:52.870116 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.809656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43321 (local address 127.0.97.3:47531)
0925 10:32:52.818791 (+ 9135us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:52.818865 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:52.822108 (+ 3243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:52.840214 (+ 18106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:52.840219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:52.840722 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:52.842340 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.842352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.849664 (+ 7312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.849669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.856502 (+ 6833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.856516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.857924 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.858002 (+ 78us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.858234 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.859198 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:52.859218 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:52.862841 (+ 3623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:52.866667 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:52.866671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:52.866675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:52.867182 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:52.867625 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:52.867629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:52.867632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:52.867773 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:52.868384 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:52.868392 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:52.869339 (+ 947us) client_negotiation.cc:769] Sending connection context
0925 10:32:52.869475 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 10:32:52.869727 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8740,"mutex_wait_us":1228,"thread_start_us":28460,"threads_started":1}
I20260925 10:32:52.871230 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.809006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47531 (local address 127.0.97.61:43321)
0925 10:32:52.839745 (+ 30739us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:52.839750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:52.839769 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:32:52.839839 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:52.839846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:52.839918 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:52.840024 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:52.847945 (+ 7921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.849411 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.856688 (+ 7277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.857296 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.857406 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.863054 (+ 5648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:52.863062 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:52.863067 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:52.863112 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:52.866519 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.867335 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.867342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.867347 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.867497 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.867925 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.867929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.867932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.868244 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:52.870484 (+ 2240us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:52.870630 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 10:32:52.870884 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30638,"thread_start_us":30565,"threads_started":1}
I20260925 10:32:52.872123 7810 heartbeater.cc:347] Connected to a master server at 127.0.97.61:43321
I20260925 10:32:52.872257 7810 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:52.872710 7810 heartbeater.cc:511] Master 127.0.97.61:43321 requested a full tablet report, sending...
I20260925 10:32:52.873905 7143 ts_manager.cc:194] Registered new tserver with Master: 9ba24b6ac5a349b3b3a88387a76dba5c (127.0.97.3:37187)
I20260925 10:32:52.877620 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.842816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.0.97.61:43321)
0925 10:32:52.848225 (+ 5409us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:52.848230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:52.851320 (+ 3090us) server_negotiation.cc:407] Connection header received
0925 10:32:52.851377 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:52.851383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:52.851473 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:52.851588 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:52.854335 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.855693 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.859247 (+ 3554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.859878 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.859963 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.870135 (+ 10172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:52.870142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:52.870149 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:52.870190 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:52.874692 (+ 4502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.875616 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.875624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.875627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.875779 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.876162 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.876169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.876171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.876526 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:52.876761 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:52.877064 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 10:32:52.877251 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5258,"thread_start_us":5108,"threads_started":1}
I20260925 10:32:52.877766 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.809656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.3:33867)
0925 10:32:52.836134 (+ 26478us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:52.836162 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:52.836353 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:52.839620 (+ 3267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:52.839631 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:52.841450 (+ 1819us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:52.842021 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.842037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.857787 (+ 15750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.857793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.859455 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.859468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.861491 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.861499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.861690 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.862429 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:52.862451 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:52.867600 (+ 5149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:52.872914 (+ 5314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:52.872918 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:52.872922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:52.873421 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:52.874752 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:52.874762 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:52.874767 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:52.874928 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:52.875638 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:52.875644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:52.875825 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 10:32:52.875986 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 10:32:52.877237 (+ 1251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26278,"mutex_wait_us":31,"spinlock_wait_cycles":44288,"thread_start_us":12535,"threads_started":1}
I20260925 10:32:52.878407 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.808973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33867 (local address 127.0.97.62:46413)
0925 10:32:52.839084 (+ 30111us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:52.839090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:52.839106 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:32:52.839164 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:52.839169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:52.839254 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:52.839361 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:52.853874 (+ 14513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.855096 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.859617 (+ 4521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.860219 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.860325 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.867816 (+ 7491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:52.867823 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:52.867828 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:52.867869 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:52.872730 (+ 4861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.874493 (+ 1763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.874498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.874502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.874626 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.875065 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.875071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.875074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.875456 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:52.875924 (+ 468us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:52.876498 (+ 574us) server_negotiation.cc:299] Negotiation successful
0925 10:32:52.877148 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29988,"thread_start_us":29918,"threads_started":1}
I20260925 10:32:52.879163 7809 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46413
I20260925 10:32:52.879295 7809 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:52.879648 7809 heartbeater.cc:511] Master 127.0.97.62:46413 requested a full tablet report, sending...
I20260925 10:32:52.880891 6546 ts_manager.cc:194] Re-registered known tserver with Master: 9ba24b6ac5a349b3b3a88387a76dba5c (127.0.97.3:37187)
I20260925 10:32:52.881862 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33867
I20260925 10:32:52.882853 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.851184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.0.97.60:36757)
0925 10:32:52.851802 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:52.851807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:52.852510 (+ 703us) server_negotiation.cc:407] Connection header received
0925 10:32:52.852737 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:52.852746 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:52.852851 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:52.852928 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:52.855977 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.857278 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.865779 (+ 8501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.866510 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.866604 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.875726 (+ 9122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:52.875734 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:52.875740 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:52.875775 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:52.879955 (+ 4180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.880802 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.880808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.880811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.880948 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.881418 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.881439 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.881444 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.881703 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:52.881965 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:52.882262 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 10:32:52.882515 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":362,"threads_started":1}
I20260925 10:32:52.885973 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.842583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.0.97.62:46413)
0925 10:32:52.858871 (+ 16288us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:52.858877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:52.858895 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:32:52.858942 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:52.858948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:52.859022 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:52.860564 (+ 1542us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:52.862772 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.866284 (+ 3512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.868725 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.869442 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.869596 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.873629 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:52.873635 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:52.873639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:52.873685 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:52.881346 (+ 7661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.884306 (+ 2960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.884312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.884316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.884460 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:52.884803 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:52.884842 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:52.884844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:52.885074 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:52.885247 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:52.885469 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 10:32:52.885620 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16163,"thread_start_us":20012,"threads_started":1}
I20260925 10:32:52.889840 6546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:32:52.897105 6546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:32:52.924129 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.913953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33123 (local address 127.0.0.1:42634)
0925 10:32:52.914949 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:52.914972 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:52.915096 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:52.916541 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:52.916550 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:52.916586 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:52.917013 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.917029 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.918265 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.918273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.920530 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.920543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.922734 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.922740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.923376 (+ 636us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.923419 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 10:32:52.923521 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:32:52.923673 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":719,"threads_started":1}
I20260925 10:32:52.924988 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.914750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.0.97.1:33123)
0925 10:32:52.916037 (+ 1287us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:52.916045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:52.916074 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:32:52.916161 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:52.916167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:52.916360 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:52.916513 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:52.917182 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.918131 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.920723 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.922215 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.922356 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.922592 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:52.924530 (+ 1938us) server_negotiation.cc:299] Negotiation successful
0925 10:32:52.924695 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1129,"thread_start_us":1069,"threads_started":1}
I20260925 10:32:52.927980 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.914743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34346 (local address 127.0.97.3:37187)
0925 10:32:52.915670 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:52.915682 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:52.918867 (+ 3185us) server_negotiation.cc:407] Connection header received
0925 10:32:52.919126 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:52.919133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:52.919387 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:52.919543 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:52.920577 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.922090 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.925236 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.926605 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.926702 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.926894 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:52.927501 (+ 607us) server_negotiation.cc:299] Negotiation successful
0925 10:32:52.927660 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":572,"threads_started":1}
I20260925 10:32:52.928321 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.915412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36132 (local address 127.0.97.2:33031)
0925 10:32:52.915990 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:52.916002 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:52.919561 (+ 3559us) server_negotiation.cc:407] Connection header received
0925 10:32:52.919656 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:52.919665 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:52.919888 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:52.920076 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:52.920763 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.921861 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.925667 (+ 3806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.927115 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.927241 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.927416 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:52.927880 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 10:32:52.928061 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":256,"threads_started":1}
I20260925 10:32:52.928635 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.915011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33031 (local address 127.0.0.1:36132)
0925 10:32:52.919294 (+ 4283us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:52.919309 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:52.919415 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:52.920087 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:52.920092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:52.920116 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:52.920620 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.920632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.922176 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.922180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.925533 (+ 3353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.925544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.927536 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.927544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.927700 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.927730 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 10:32:52.927828 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 10:32:52.927943 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4178,"mutex_wait_us":536,"thread_start_us":3051,"threads_started":1}
I20260925 10:32:52.928793 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:52.915011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37187 (local address 127.0.0.1:34346)
0925 10:32:52.918751 (+ 3740us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:52.918765 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:52.918900 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:52.919881 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:52.919885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:52.919909 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:52.920365 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.920375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.922244 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.922252 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.925053 (+ 2801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:52.925067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:52.927077 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:52.927083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:52.927257 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:52.927326 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 10:32:52.928210 (+ 884us) client_negotiation.cc:240] Negotiation successful
0925 10:32:52.928340 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3627,"mutex_wait_us":476,"thread_start_us":4445,"threads_started":1}
I20260925 10:32:52.929019 7454 tablet_service.cc:1467] Processing CreateTablet for tablet c282380b0c7a4639a531bacabba66737 (DEFAULT_TABLE table=test-table [id=be8129e5f5b1440c9c9035ecc8125ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:52.930558 7454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c282380b0c7a4639a531bacabba66737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:52.932519 7602 tablet_service.cc:1467] Processing CreateTablet for tablet c282380b0c7a4639a531bacabba66737 (DEFAULT_TABLE table=test-table [id=be8129e5f5b1440c9c9035ecc8125ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:52.932348 7743 tablet_service.cc:1467] Processing CreateTablet for tablet c282380b0c7a4639a531bacabba66737 (DEFAULT_TABLE table=test-table [id=be8129e5f5b1440c9c9035ecc8125ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:52.933538 7602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c282380b0c7a4639a531bacabba66737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:52.933684 7743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c282380b0c7a4639a531bacabba66737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:52.946163 7842 tablet_bootstrap.cc:492] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88: Bootstrap starting.
I20260925 10:32:52.949002 7842 tablet_bootstrap.cc:654] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:52.950027 7844 tablet_bootstrap.cc:492] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c: Bootstrap starting.
I20260925 10:32:52.950929 7843 tablet_bootstrap.cc:492] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba: Bootstrap starting.
I20260925 10:32:52.954123 7844 tablet_bootstrap.cc:654] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:52.954407 7842 log.cc:824] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:52.954818 7843 tablet_bootstrap.cc:654] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:52.955408 7844 log.cc:824] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:52.956025 7843 log.cc:824] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:52.957883 7842 tablet_bootstrap.cc:492] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88: No bootstrap required, opened a new log
I20260925 10:32:52.958221 7842 ts_tablet_manager.cc:1397] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260925 10:32:52.958274 7844 tablet_bootstrap.cc:492] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c: No bootstrap required, opened a new log
I20260925 10:32:52.958556 7844 ts_tablet_manager.cc:1397] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 10:32:52.959115 7843 tablet_bootstrap.cc:492] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba: No bootstrap required, opened a new log
I20260925 10:32:52.959424 7843 ts_tablet_manager.cc:1397] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260925 10:32:52.965440 7844 raft_consensus.cc:348] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } } peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } }
I20260925 10:32:52.965801 7844 raft_consensus.cc:374] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:52.965893 7844 raft_consensus.cc:729] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba24b6ac5a349b3b3a88387a76dba5c, State: Initialized, Role: FOLLOWER
I20260925 10:32:52.966559 7844 consensus_queue.cc:260] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } } peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } }
I20260925 10:32:52.968657 7844 ts_tablet_manager.cc:1428] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260925 10:32:52.968919 7809 heartbeater.cc:503] Master 127.0.97.62:46413 was elected leader, sending a full tablet report...
I20260925 10:32:52.970443 7843 raft_consensus.cc:348] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } } peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } }
I20260925 10:32:52.970858 7843 raft_consensus.cc:374] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:52.970940 7843 raft_consensus.cc:729] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65690ad1c05d4a8083522828aca777ba, State: Initialized, Role: FOLLOWER
I20260925 10:32:52.970804 7842 raft_consensus.cc:348] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } } peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } }
I20260925 10:32:52.971195 7842 raft_consensus.cc:374] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:52.971272 7842 raft_consensus.cc:729] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba74a37e8f444618ecb68f3c6a44c88, State: Initialized, Role: FOLLOWER
I20260925 10:32:52.971563 7843 consensus_queue.cc:260] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } } peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } }
I20260925 10:32:52.971902 7842 consensus_queue.cc:260] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } } peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } }
I20260925 10:32:52.976666 7843 ts_tablet_manager.cc:1428] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba: Time spent starting tablet: real 0.017s user 0.015s sys 0.002s
I20260925 10:32:52.977218 7668 heartbeater.cc:503] Master 127.0.97.62:46413 was elected leader, sending a full tablet report...
I20260925 10:32:52.980014 7842 ts_tablet_manager.cc:1428] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88: Time spent starting tablet: real 0.022s user 0.012s sys 0.005s
W20260925 10:32:52.985283 7670 tablet.cc:2369] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:32:53.030604 7849 raft_consensus.cc:482] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:32:53.030947 7849 raft_consensus.cc:504] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } } peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } }
I20260925 10:32:53.032593 7849 leader_election.cc:288] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ba24b6ac5a349b3b3a88387a76dba5c (127.0.97.3:37187), 2ba74a37e8f444618ecb68f3c6a44c88 (127.0.97.2:33031)
I20260925 10:32:53.041415 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:53.033052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56455 (local address 127.0.97.2:33031)
0925 10:32:53.033268 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:53.033275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:53.033721 (+ 446us) server_negotiation.cc:407] Connection header received
0925 10:32:53.033950 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:53.033956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:53.034031 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:53.034133 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:53.034956 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.036074 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.039204 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.040446 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.040581 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:53.040677 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:53.041065 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 10:32:53.041159 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:32:53.041674 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:53.032813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33031 (local address 127.0.97.1:56455)
0925 10:32:53.033593 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:53.033614 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:53.033782 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:53.034181 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:53.034186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:53.034229 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:53.034804 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:53.034812 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.036213 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.036217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:53.039019 (+ 2802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:53.039034 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.040674 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.040680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:53.040820 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:53.040848 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 10:32:53.041001 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 10:32:53.041164 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":222,"spinlock_wait_cycles":7552,"thread_start_us":634,"threads_started":1}
I20260925 10:32:53.042135 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:53.033011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40049 (local address 127.0.97.3:37187)
0925 10:32:53.033244 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:53.033250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:53.033885 (+ 635us) server_negotiation.cc:407] Connection header received
0925 10:32:53.033966 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:53.033970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:53.034033 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:53.034133 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:53.034952 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.036074 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.039204 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.040453 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.040581 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:53.040681 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:53.041693 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0925 10:32:53.041864 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:32:53.042357 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:53.032817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37187 (local address 127.0.97.1:40049)
0925 10:32:53.033758 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:53.033772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:53.033874 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:53.034181 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:53.034186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:53.034227 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:53.034782 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:53.034799 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.036211 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.036217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:53.039020 (+ 2803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:53.039034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.041092 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.041099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:53.041270 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:53.041330 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 10:32:53.041419 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:32:53.041550 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":319,"spinlock_wait_cycles":8448,"thread_start_us":324,"threads_started":1}
I20260925 10:32:53.042857 7622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c282380b0c7a4639a531bacabba66737" candidate_uuid: "65690ad1c05d4a8083522828aca777ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" is_pre_election: true
I20260925 10:32:53.043445 7763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c282380b0c7a4639a531bacabba66737" candidate_uuid: "65690ad1c05d4a8083522828aca777ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" is_pre_election: true
I20260925 10:32:53.043535 7622 raft_consensus.cc:2454] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65690ad1c05d4a8083522828aca777ba in term 0.
I20260925 10:32:53.044427 7763 raft_consensus.cc:2454] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65690ad1c05d4a8083522828aca777ba in term 0.
I20260925 10:32:53.044723 7410 leader_election.cc:302] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba74a37e8f444618ecb68f3c6a44c88, 65690ad1c05d4a8083522828aca777ba; no voters:
I20260925 10:32:53.045192 7849 raft_consensus.cc:2781] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:32:53.045389 7849 raft_consensus.cc:482] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:32:53.045511 7849 raft_consensus.cc:3037] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:53.049461 7849 raft_consensus.cc:504] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } } peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } }
I20260925 10:32:53.050276 7849 leader_election.cc:288] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [CANDIDATE]: Term 1 election: Requested vote from peers 9ba24b6ac5a349b3b3a88387a76dba5c (127.0.97.3:37187), 2ba74a37e8f444618ecb68f3c6a44c88 (127.0.97.2:33031)
I20260925 10:32:53.050647 7763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c282380b0c7a4639a531bacabba66737" candidate_uuid: "65690ad1c05d4a8083522828aca777ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c"
I20260925 10:32:53.050858 7763 raft_consensus.cc:3037] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:53.050882 7622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c282380b0c7a4639a531bacabba66737" candidate_uuid: "65690ad1c05d4a8083522828aca777ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ba74a37e8f444618ecb68f3c6a44c88"
I20260925 10:32:53.051087 7622 raft_consensus.cc:3037] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:53.054452 7763 raft_consensus.cc:2454] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65690ad1c05d4a8083522828aca777ba in term 1.
I20260925 10:32:53.054452 7622 raft_consensus.cc:2454] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65690ad1c05d4a8083522828aca777ba in term 1.
I20260925 10:32:53.054879 7408 leader_election.cc:302] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65690ad1c05d4a8083522828aca777ba, 9ba24b6ac5a349b3b3a88387a76dba5c; no voters:
I20260925 10:32:53.055207 7849 raft_consensus.cc:2781] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:53.055720 7849 raft_consensus.cc:686] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [term 1 LEADER]: Becoming Leader. State: Replica: 65690ad1c05d4a8083522828aca777ba, State: Running, Role: LEADER
I20260925 10:32:53.056244 7849 consensus_queue.cc:237] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } } peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } }
W20260925 10:32:53.058880 7811 tablet.cc:2369] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:32:53.061640 6546 catalog_manager.cc:5495] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 65690ad1c05d4a8083522828aca777ba (127.0.97.1). New cstate: current_term: 1 leader_uuid: "65690ad1c05d4a8083522828aca777ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } health_report { overall_health: UNKNOWN } } }
I20260925 10:32:53.079834 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:53.074685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.0.97.1:33123)
0925 10:32:53.074878 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:53.074882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:53.074898 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:32:53.075042 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:53.075045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:53.075110 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:53.075230 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:32:53.075528 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.076272 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.077644 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.078107 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.078203 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:53.078733 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:32:53.079256 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:32:53.079453 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:53.079505 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 10:32:53.079600 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:32:53.087513 7763 raft_consensus.cc:1261] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 1 FOLLOWER]: Refusing update from remote peer 65690ad1c05d4a8083522828aca777ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:53.087508 7622 raft_consensus.cc:1261] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 1 FOLLOWER]: Refusing update from remote peer 65690ad1c05d4a8083522828aca777ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:53.088413 7853 consensus_queue.cc:1035] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:53.088757 7849 consensus_queue.cc:1035] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:53.103921 7858 mvcc.cc:205] Tried to move back new op lower bound from 7333201400156831744 to 7333201400042168320. Current Snapshot: MvccSnapshot[applied={T|T < 7333201400156831744}]
I20260925 10:32:53.120610 6545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:53.120925 6545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:53.127977 6545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8edbe7ca91c47a2ac5daed766a95286: Sending DeleteTablet for 3 replicas of tablet c282380b0c7a4639a531bacabba66737
I20260925 10:32:53.128952 7743 tablet_service.cc:1514] Processing DeleteTablet for tablet c282380b0c7a4639a531bacabba66737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34346
I20260925 10:32:53.129335 7454 tablet_service.cc:1514] Processing DeleteTablet for tablet c282380b0c7a4639a531bacabba66737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42634
I20260925 10:32:53.129446 7602 tablet_service.cc:1514] Processing DeleteTablet for tablet c282380b0c7a4639a531bacabba66737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36132
I20260925 10:32:53.130259 7867 tablet_replica.cc:330] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba: stopping tablet replica
I20260925 10:32:53.130831 7867 raft_consensus.cc:2229] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:32:53.131625 7867 raft_consensus.cc:2258] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:53.141549 7867 ts_tablet_manager.cc:1905] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:53.149756 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 6514
I20260925 10:32:53.150964 6576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:53.158516 7866 tablet_replica.cc:330] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c: stopping tablet replica
I20260925 10:32:53.159070 7866 raft_consensus.cc:2229] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:53.160121 7866 raft_consensus.cc:2258] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:53.160921 7868 tablet_replica.cc:330] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88: stopping tablet replica
I20260925 10:32:53.161275 7868 raft_consensus.cc:2229] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:53.161669 7868 raft_consensus.cc:2258] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:53.165374 7868 ts_tablet_manager.cc:1905] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:53.167135 7866 ts_tablet_manager.cc:1905] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:53.170733 7867 ts_tablet_manager.cc:1918] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:32:53.170992 7867 log.cc:1196] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/c282380b0c7a4639a531bacabba66737
I20260925 10:32:53.171738 7867 ts_tablet_manager.cc:1939] T c282380b0c7a4639a531bacabba66737 P 65690ad1c05d4a8083522828aca777ba: Deleting consensus metadata
I20260925 10:32:53.178248 7866 ts_tablet_manager.cc:1918] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:32:53.178529 7866 log.cc:1196] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/c282380b0c7a4639a531bacabba66737
I20260925 10:32:53.179215 7866 ts_tablet_manager.cc:1939] T c282380b0c7a4639a531bacabba66737 P 9ba24b6ac5a349b3b3a88387a76dba5c: Deleting consensus metadata
I20260925 10:32:53.183257 7868 ts_tablet_manager.cc:1918] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:32:53.183439 7868 log.cc:1196] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/c282380b0c7a4639a531bacabba66737
I20260925 10:32:53.183879 7868 ts_tablet_manager.cc:1939] T c282380b0c7a4639a531bacabba66737 P 2ba74a37e8f444618ecb68f3c6a44c88: Deleting consensus metadata
I20260925 10:32:53.186323 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:53.132084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60512 (local address 127.0.97.61:43321)
0925 10:32:53.132264 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:53.132273 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:53.147761 (+ 15488us) server_negotiation.cc:407] Connection header received
0925 10:32:53.151075 (+ 3314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:53.151084 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:53.151192 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:53.151279 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:53.156241 (+ 4962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.157709 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.167936 (+ 10227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.168577 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.168677 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:53.172925 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:53.172931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:53.172937 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:53.172978 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:53.176903 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:53.178451 (+ 1548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:53.178456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:53.178459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:53.178609 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:53.179400 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:53.179408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:53.179414 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:53.179752 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:53.179973 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:53.185712 (+ 5739us) server_negotiation.cc:299] Negotiation successful
0925 10:32:53.185959 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:32:53.200103 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:53.148388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.0.97.60:36757)
0925 10:32:53.148624 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:53.148632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:53.186484 (+ 37852us) server_negotiation.cc:407] Connection header received
0925 10:32:53.187640 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:53.187646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:53.187729 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:53.187801 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:53.189230 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.189993 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.191303 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.191702 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.191853 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:53.194392 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:53.194398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:53.194401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:53.194430 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:53.196996 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:53.197660 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:53.197666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:53.197669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:53.197810 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:53.198225 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:53.198230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:53.198233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:53.198627 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:53.198883 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:53.199157 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 10:32:53.199362 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:32:53.353271 6531 catalog_manager.cc:4841] TS 9ba24b6ac5a349b3b3a88387a76dba5c (127.0.97.3:37187): tablet c282380b0c7a4639a531bacabba66737 (table test-table [id=be8129e5f5b1440c9c9035ecc8125ade]) successfully deleted
I20260925 10:32:53.354743 6533 catalog_manager.cc:4841] TS 2ba74a37e8f444618ecb68f3c6a44c88 (127.0.97.2:33031): tablet c282380b0c7a4639a531bacabba66737 (table test-table [id=be8129e5f5b1440c9c9035ecc8125ade]) successfully deleted
I20260925 10:32:53.354990 6533 catalog_manager.cc:4841] TS 65690ad1c05d4a8083522828aca777ba (127.0.97.1:33123): tablet c282380b0c7a4639a531bacabba66737 (table test-table [id=be8129e5f5b1440c9c9035ecc8125ade]) successfully deleted
I20260925 10:32:53.355362 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 6514
I20260925 10:32:53.414604 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:53.382266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.0.97.61:43321)
0925 10:32:53.382475 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:53.382482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:53.398867 (+ 16385us) server_negotiation.cc:407] Connection header received
0925 10:32:53.400222 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:53.400246 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:53.400321 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:53.400384 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:53.402606 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.403685 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.405596 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.406236 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.406359 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:53.409076 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:53.409083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:53.409087 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:53.409125 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:53.411715 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:53.412459 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:53.412464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:53.412468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:53.412589 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:53.412972 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:53.412977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:53.412981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:53.413391 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:53.413615 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:53.414003 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 10:32:53.414210 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 10:32:53.416652 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:53.399672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.0.97.60:36757)
0925 10:32:53.399833 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:53.399842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:53.399865 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:32:53.400087 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:53.400092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:53.400189 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:53.400318 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:53.402567 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.403723 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.405864 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:53.406544 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:53.406640 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:53.409865 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:53.409871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:53.409875 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:53.409915 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:53.413389 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:53.414209 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:53.414215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:53.414219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:53.414389 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:53.415144 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:53.415149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:53.415152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:53.415457 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:53.415669 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:53.416264 (+ 595us) server_negotiation.cc:299] Negotiation successful
0925 10:32:53.416417 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:32:54.113962 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:54.113069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.3:55333)
0925 10:32:54.113572 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:54.113724 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":227,"threads_started":1}
I20260925 10:32:54.114194 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:54.113327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.1:44423)
0925 10:32:54.113846 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:54.113967 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":290,"threads_started":1}
W20260925 10:32:54.114535 7809 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46413 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
W20260925 10:32:54.114655 7520 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46413 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
I20260925 10:32:54.117977 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:54.117203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.2:54721)
0925 10:32:54.117649 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:54.117777 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":232,"threads_started":1}
W20260925 10:32:54.118434 7668 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46413 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
I20260925 10:32:54.627708 7880 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e8edbe7ca91c47a2ac5daed766a95286)
I20260925 10:32:54.628057 7880 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } }
I20260925 10:32:54.628279 7881 raft_consensus.cc:482] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e8edbe7ca91c47a2ac5daed766a95286)
I20260925 10:32:54.628545 7881 raft_consensus.cc:504] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } }
I20260925 10:32:54.629457 7880 leader_election.cc:288] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8edbe7ca91c47a2ac5daed766a95286 (127.0.97.62:46413), e36414eb5d1d42b6a5461ee76d162ca6 (127.0.97.60:36757)
I20260925 10:32:54.630569 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:54.629682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:48662)
0925 10:32:54.630115 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:54.630275 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":191,"threads_started":1}
I20260925 10:32:54.631302 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:54.629901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:48664)
0925 10:32:54.630826 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:54.631004 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":682,"threads_started":1}
W20260925 10:32:54.631552 7128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
I20260925 10:32:54.631692 7881 leader_election.cc:288] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8edbe7ca91c47a2ac5daed766a95286 (127.0.97.62:46413), ca8ca5ced019409bacfa9b2e62f58a58 (127.0.97.61:43321)
W20260925 10:32:54.631793 7324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
I20260925 10:32:54.633589 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:54.633058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:48666)
0925 10:32:54.633239 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:54.633371 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 10:32:54.633765 7128 leader_election.cc:334] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e8edbe7ca91c47a2ac5daed766a95286 (127.0.97.62:46413): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
I20260925 10:32:54.634358 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:54.633282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:48668)
0925 10:32:54.633832 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:54.633994 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":289,"threads_started":1}
W20260925 10:32:54.634591 7324 leader_election.cc:334] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e8edbe7ca91c47a2ac5daed766a95286 (127.0.97.62:46413): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
I20260925 10:32:54.639122 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:54.630377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36757 (local address 127.0.0.1:55606)
0925 10:32:54.630864 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:54.630881 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:54.631027 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:54.632505 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:54.632509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:54.632537 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:54.632826 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:54.632844 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:54.635738 (+ 2894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:54.635743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:54.637180 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:54.637191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:54.638333 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:54.638340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:54.638538 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:54.638581 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 10:32:54.638704 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 10:32:54.638849 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":847,"threads_started":1}
I20260925 10:32:54.639189 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:54.630869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.0.97.60:36757)
0925 10:32:54.632118 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:54.632126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:54.632149 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:32:54.632217 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:54.632223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:54.632312 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:54.632983 (+ 671us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:54.634212 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:54.635587 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:54.637321 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:54.638146 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:54.638286 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:54.638503 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:54.638728 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 10:32:54.638854 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":836,"thread_start_us":705,"threads_started":1}
I20260925 10:32:54.639587 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:54.632560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.0.97.61:43321)
0925 10:32:54.633783 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:54.633790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:54.633812 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:32:54.633867 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:54.633872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:54.633946 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:54.634043 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:54.634911 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:54.636040 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:54.637691 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:54.638579 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:54.638732 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:54.638841 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:54.639131 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 10:32:54.639272 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":542,"threads_started":1}
I20260925 10:32:54.639855 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:54.632168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43321 (local address 127.0.0.1:60528)
0925 10:32:54.633325 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:54.633343 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:54.633464 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:54.634092 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:54.634098 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:54.634133 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:54.634612 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:54.634625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:54.636201 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:54.636206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:54.637534 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:54.637545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:54.638769 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:54.638775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:54.638941 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:54.638967 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 10:32:54.639106 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 10:32:54.639510 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"mutex_wait_us":23}
I20260925 10:32:54.639816 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" is_pre_election: true
I20260925 10:32:54.640157 7349 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8ca5ced019409bacfa9b2e62f58a58 in term 1.
I20260925 10:32:54.640429 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" is_pre_election: true
I20260925 10:32:54.640667 7128 leader_election.cc:302] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ca8ca5ced019409bacfa9b2e62f58a58, e36414eb5d1d42b6a5461ee76d162ca6; no voters: e8edbe7ca91c47a2ac5daed766a95286
I20260925 10:32:54.640723 7153 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e36414eb5d1d42b6a5461ee76d162ca6 in term 1.
I20260925 10:32:54.641017 7880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 10:32:54.641150 7880 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e8edbe7ca91c47a2ac5daed766a95286)
I20260925 10:32:54.641249 7880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:32:54.641222 7326 leader_election.cc:302] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ca8ca5ced019409bacfa9b2e62f58a58, e36414eb5d1d42b6a5461ee76d162ca6; no voters: e8edbe7ca91c47a2ac5daed766a95286
I20260925 10:32:54.641598 7881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 10:32:54.641714 7881 raft_consensus.cc:482] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e8edbe7ca91c47a2ac5daed766a95286)
I20260925 10:32:54.641811 7881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:32:54.643460 7880 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } }
I20260925 10:32:54.643841 7881 raft_consensus.cc:504] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } }
I20260925 10:32:54.644094 7880 leader_election.cc:288] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [CANDIDATE]: Term 2 election: Requested vote from peers e8edbe7ca91c47a2ac5daed766a95286 (127.0.97.62:46413), e36414eb5d1d42b6a5461ee76d162ca6 (127.0.97.60:36757)
I20260925 10:32:54.644474 7881 leader_election.cc:288] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [CANDIDATE]: Term 2 election: Requested vote from peers e8edbe7ca91c47a2ac5daed766a95286 (127.0.97.62:46413), ca8ca5ced019409bacfa9b2e62f58a58 (127.0.97.61:43321)
I20260925 10:32:54.644546 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:54.644204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:48680)
0925 10:32:54.644341 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:54.644421 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 10:32:54.644887 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e36414eb5d1d42b6a5461ee76d162ca6"
I20260925 10:32:54.645185 7349 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ca8ca5ced019409bacfa9b2e62f58a58 in current term 2: Already voted for candidate e36414eb5d1d42b6a5461ee76d162ca6 in this term.
I20260925 10:32:54.645280 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ca8ca5ced019409bacfa9b2e62f58a58"
I20260925 10:32:54.645519 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:54.645057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:48688)
0925 10:32:54.645221 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:54.645349 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 10:32:54.645601 7153 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e36414eb5d1d42b6a5461ee76d162ca6 in current term 2: Already voted for candidate ca8ca5ced019409bacfa9b2e62f58a58 in this term.
W20260925 10:32:54.645776 7128 leader_election.cc:334] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e8edbe7ca91c47a2ac5daed766a95286 (127.0.97.62:46413): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
I20260925 10:32:54.645901 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:54.644529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:48682)
0925 10:32:54.644867 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:54.645730 (+ 863us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 10:32:54.645906 7128 leader_election.cc:302] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca8ca5ced019409bacfa9b2e62f58a58; no voters: e36414eb5d1d42b6a5461ee76d162ca6, e8edbe7ca91c47a2ac5daed766a95286
I20260925 10:32:54.646143 7880 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 10:32:54.646870 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:54.646608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:48700)
0925 10:32:54.646717 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:54.646784 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 10:32:54.647039 7324 leader_election.cc:334] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e8edbe7ca91c47a2ac5daed766a95286 (127.0.97.62:46413): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
I20260925 10:32:54.647158 7324 leader_election.cc:302] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36414eb5d1d42b6a5461ee76d162ca6; no voters: ca8ca5ced019409bacfa9b2e62f58a58, e8edbe7ca91c47a2ac5daed766a95286
I20260925 10:32:54.647423 7881 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 10:32:55.116551 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:55.115196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.1:49439)
0925 10:32:55.116253 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:55.116351 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":891,"thread_start_us":763,"threads_started":1}
I20260925 10:32:55.116730 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:55.115194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.3:48121)
0925 10:32:55.116424 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:55.116543 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1108,"thread_start_us":992,"threads_started":1}
I20260925 10:32:55.119685 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:55.119045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.2:36399)
0925 10:32:55.119426 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:55.119493 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":216,"threads_started":1}
I20260925 10:32:56.118369 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.117477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.3:46937)
0925 10:32:56.118021 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.118120 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":303,"threads_started":1}
I20260925 10:32:56.118351 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.117481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.1:45811)
0925 10:32:56.118036 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.118120 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":353,"threads_started":1}
I20260925 10:32:56.121173 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.120447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.2:44029)
0925 10:32:56.120878 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.121033 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":236,"threads_started":1}
I20260925 10:32:56.409725 7896 raft_consensus.cc:482] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:32:56.409905 7896 raft_consensus.cc:504] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } }
I20260925 10:32:56.410605 7896 leader_election.cc:288] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e8edbe7ca91c47a2ac5daed766a95286 (127.0.97.62:46413), ca8ca5ced019409bacfa9b2e62f58a58 (127.0.97.61:43321)
I20260925 10:32:56.411250 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" is_pre_election: true
I20260925 10:32:56.411502 7153 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e36414eb5d1d42b6a5461ee76d162ca6 in term 2.
I20260925 10:32:56.411649 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.410752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:48926)
0925 10:32:56.411291 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.411422 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":234,"threads_started":1}
I20260925 10:32:56.411916 7326 leader_election.cc:302] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca8ca5ced019409bacfa9b2e62f58a58, e36414eb5d1d42b6a5461ee76d162ca6; no voters:
I20260925 10:32:56.412182 7896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 10:32:56.412420 7896 raft_consensus.cc:482] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:32:56.412549 7896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:32:56.413169 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.412877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:48932)
0925 10:32:56.412992 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.413054 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 10:32:56.413367 7324 leader_election.cc:334] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e8edbe7ca91c47a2ac5daed766a95286 (127.0.97.62:46413): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
I20260925 10:32:56.414847 7896 raft_consensus.cc:504] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } }
I20260925 10:32:56.415416 7896 leader_election.cc:288] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [CANDIDATE]: Term 3 election: Requested vote from peers e8edbe7ca91c47a2ac5daed766a95286 (127.0.97.62:46413), ca8ca5ced019409bacfa9b2e62f58a58 (127.0.97.61:43321)
I20260925 10:32:56.415900 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.415475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:48934)
0925 10:32:56.415712 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.415790 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260925 10:32:56.415889 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ca8ca5ced019409bacfa9b2e62f58a58"
I20260925 10:32:56.416098 7153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:32:56.416780 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.416533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:48948)
0925 10:32:56.416641 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.416696 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 10:32:56.416971 7324 leader_election.cc:334] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e8edbe7ca91c47a2ac5daed766a95286 (127.0.97.62:46413): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
I20260925 10:32:56.418370 7153 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e36414eb5d1d42b6a5461ee76d162ca6 in term 3.
I20260925 10:32:56.418699 7326 leader_election.cc:302] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ca8ca5ced019409bacfa9b2e62f58a58, e36414eb5d1d42b6a5461ee76d162ca6; no voters: e8edbe7ca91c47a2ac5daed766a95286
I20260925 10:32:56.418943 7896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 10:32:56.419302 7896 raft_consensus.cc:686] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [term 3 LEADER]: Becoming Leader. State: Replica: e36414eb5d1d42b6a5461ee76d162ca6, State: Running, Role: LEADER
I20260925 10:32:56.419655 7896 consensus_queue.cc:237] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } }
I20260925 10:32:56.421141 7899 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e36414eb5d1d42b6a5461ee76d162ca6. Latest consensus state: current_term: 3 leader_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } } }
I20260925 10:32:56.421445 7899 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:56.422240 7901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:56.426719 7901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:56.427657 7901 catalog_manager.cc:1259] Loaded cluster ID: 04db75e16edb4e8fa187793f7336f08a
I20260925 10:32:56.427774 7901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:56.428997 7901 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:56.430346 7901 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: Loaded TSK: 0
I20260925 10:32:56.431172 7901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:32:56.520706 7339 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:32:56.527889 7339 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:32:56.531956 7153 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [term 3 FOLLOWER]: Refusing update from remote peer e36414eb5d1d42b6a5461ee76d162ca6: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 10:32:56.532024 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.531544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:48982)
0925 10:32:56.531754 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.531844 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260925 10:32:56.532742 7899 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 10:32:56.532990 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.532667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:48988)
0925 10:32:56.532801 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.532865 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 10:32:56.533195 7324 consensus_peers.cc:487] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 -> Peer e8edbe7ca91c47a2ac5daed766a95286 (127.0.97.62:46413): Couldn't send request to peer e8edbe7ca91c47a2ac5daed766a95286. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 10:32:56.535836 7903 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e36414eb5d1d42b6a5461ee76d162ca6. Latest consensus state: current_term: 3 leader_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } } }
I20260925 10:32:56.536104 7903 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:32:56.536722 7899 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } } }
I20260925 10:32:56.537014 7899 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:56.540194 7903 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e8edbe7ca91c47a2ac5daed766a95286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46413 } } peers { permanent_uuid: "ca8ca5ced019409bacfa9b2e62f58a58" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43321 } attrs { promote: false } } peers { permanent_uuid: "e36414eb5d1d42b6a5461ee76d162ca6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36757 } attrs { promote: false } } }
I20260925 10:32:56.540426 7903 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca8ca5ced019409bacfa9b2e62f58a58 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:32:56.552662 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.544648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33031 (local address 127.0.0.1:36144)
0925 10:32:56.544805 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.544823 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:56.544962 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:56.547612 (+ 2650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:56.547617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:56.547645 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:56.547898 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:56.547906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:56.548872 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:56.548880 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:56.550274 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:56.550285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:56.551739 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:56.551746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:56.552005 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:56.552040 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:32:56.552141 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 10:32:56.552292 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":22}
I20260925 10:32:56.552709 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.544780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.0.97.2:33031)
0925 10:32:56.547321 (+ 2541us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:56.547327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:56.547345 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:32:56.547411 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:56.547416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:56.547488 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:56.547588 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:56.548022 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:56.548751 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:56.550522 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:56.551555 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:56.551709 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:56.551814 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:56.552165 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 10:32:56.552305 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2396,"thread_start_us":2431,"threads_started":1}
I20260925 10:32:56.554431 7602 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3bf5a562034c82a9e2035d712eeb17 (DEFAULT_TABLE table=test-table [id=642c8a66edaf4e328acafeee72de9c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:56.554877 7602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3bf5a562034c82a9e2035d712eeb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:56.557610 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.545066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37187 (local address 127.0.0.1:34348)
0925 10:32:56.551003 (+ 5937us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.551025 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:56.551226 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:56.551778 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:56.551783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:56.551879 (+ 96us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:56.552322 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:56.552335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:56.553818 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:56.553822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:56.555447 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:56.555458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:56.556942 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:56.556946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:56.557054 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:56.557076 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 10:32:56.557144 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 10:32:56.557303 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5781,"mutex_wait_us":2,"thread_start_us":462,"threads_started":1}
I20260925 10:32:56.557643 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.545403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.0.97.3:37187)
0925 10:32:56.546998 (+ 1595us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:56.547002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:56.551120 (+ 4118us) server_negotiation.cc:407] Connection header received
0925 10:32:56.551401 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:56.551407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:56.551491 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:56.551588 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:56.552593 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:56.553713 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:56.555584 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:56.556793 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:56.556924 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:56.557021 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:56.557188 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 10:32:56.557346 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1498,"thread_start_us":1389,"threads_started":1}
I20260925 10:32:56.558357 7743 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3bf5a562034c82a9e2035d712eeb17 (DEFAULT_TABLE table=test-table [id=642c8a66edaf4e328acafeee72de9c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:56.558785 7743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3bf5a562034c82a9e2035d712eeb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:56.561383 7914 tablet_bootstrap.cc:492] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88: Bootstrap starting.
I20260925 10:32:56.563021 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.545477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33123 (local address 127.0.0.1:42644)
0925 10:32:56.551732 (+ 6255us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.551747 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:56.551854 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:56.552363 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:56.552369 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:56.552399 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:32:56.552807 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:56.552818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:56.558447 (+ 5629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:56.558452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:56.560672 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:56.560684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:56.562257 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:56.562265 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:56.562491 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:56.562521 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 10:32:56.562609 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:32:56.562721 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6127,"thread_start_us":6023,"threads_started":1}
I20260925 10:32:56.563072 7914 tablet_bootstrap.cc:654] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:56.563099 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.545660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.0.97.1:33123)
0925 10:32:56.546284 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:56.546289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:56.551813 (+ 5524us) server_negotiation.cc:407] Connection header received
0925 10:32:56.551993 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:56.552001 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:56.552077 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:56.552171 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:32:56.553025 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:56.553960 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:56.560819 (+ 6859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:56.562054 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:56.562198 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:56.562297 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:56.562664 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 10:32:56.562801 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":308,"threads_started":1}
I20260925 10:32:56.564121 7454 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3bf5a562034c82a9e2035d712eeb17 (DEFAULT_TABLE table=test-table [id=642c8a66edaf4e328acafeee72de9c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:32:56.564493 7454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3bf5a562034c82a9e2035d712eeb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:56.565187 7914 tablet_bootstrap.cc:492] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88: No bootstrap required, opened a new log
I20260925 10:32:56.565308 7914 ts_tablet_manager.cc:1397] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 10:32:56.566002 7914 raft_consensus.cc:348] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } } peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } }
I20260925 10:32:56.566206 7914 raft_consensus.cc:374] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:56.566282 7914 raft_consensus.cc:729] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba74a37e8f444618ecb68f3c6a44c88, State: Initialized, Role: FOLLOWER
I20260925 10:32:56.566558 7914 consensus_queue.cc:260] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } } peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } }
I20260925 10:32:56.567387 7914 ts_tablet_manager.cc:1428] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 10:32:56.568652 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.567894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.2:41809)
0925 10:32:56.568418 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.568520 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387}
I20260925 10:32:56.568790 7916 tablet_bootstrap.cc:492] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c: Bootstrap starting.
I20260925 10:32:56.570410 7916 tablet_bootstrap.cc:654] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:56.571768 7918 tablet_bootstrap.cc:492] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba: Bootstrap starting.
I20260925 10:32:56.572191 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.571682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.2:33061)
0925 10:32:56.571985 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.572061 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260925 10:32:56.572604 7667 heartbeater.cc:503] Master 127.0.97.60:36757 was elected leader, sending a full tablet report...
I20260925 10:32:56.573513 7918 tablet_bootstrap.cc:654] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:56.574365 7916 tablet_bootstrap.cc:492] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c: No bootstrap required, opened a new log
I20260925 10:32:56.574514 7916 ts_tablet_manager.cc:1397] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 10:32:56.575224 7916 raft_consensus.cc:348] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } } peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } }
I20260925 10:32:56.575378 7916 raft_consensus.cc:374] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:56.575390 7918 tablet_bootstrap.cc:492] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba: No bootstrap required, opened a new log
I20260925 10:32:56.575476 7916 raft_consensus.cc:729] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba24b6ac5a349b3b3a88387a76dba5c, State: Initialized, Role: FOLLOWER
I20260925 10:32:56.575522 7918 ts_tablet_manager.cc:1397] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 10:32:56.575723 7916 consensus_queue.cc:260] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } } peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } }
I20260925 10:32:56.576393 7916 ts_tablet_manager.cc:1428] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:32:56.576288 7918 raft_consensus.cc:348] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } } peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } }
I20260925 10:32:56.576618 7918 raft_consensus.cc:374] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:56.576756 7918 raft_consensus.cc:729] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65690ad1c05d4a8083522828aca777ba, State: Initialized, Role: FOLLOWER
I20260925 10:32:56.577030 7918 consensus_queue.cc:260] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } } peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } }
I20260925 10:32:56.578145 7808 heartbeater.cc:503] Master 127.0.97.60:36757 was elected leader, sending a full tablet report...
I20260925 10:32:56.582674 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.576700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.3:42891)
0925 10:32:56.582448 (+ 5748us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.582537 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5638}
I20260925 10:32:56.583685 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.583337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.3:34685)
0925 10:32:56.583468 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.583537 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 10:32:56.583774 7918 ts_tablet_manager.cc:1428] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260925 10:32:56.584645 7519 heartbeater.cc:503] Master 127.0.97.60:36757 was elected leader, sending a full tablet report...
I20260925 10:32:56.585182 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.584280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.1:39873)
0925 10:32:56.584963 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.585046 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580}
I20260925 10:32:56.586822 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.586439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.1:39339)
0925 10:32:56.586589 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.586670 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 10:32:56.603282 7522 tablet.cc:2369] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:32:56.843883 7922 raft_consensus.cc:482] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:32:56.844157 7922 raft_consensus.cc:504] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } } peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } }
I20260925 10:32:56.844832 7922 leader_election.cc:288] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ba74a37e8f444618ecb68f3c6a44c88 (127.0.97.2:33031), 9ba24b6ac5a349b3b3a88387a76dba5c (127.0.97.3:37187)
I20260925 10:32:56.845369 7622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3bf5a562034c82a9e2035d712eeb17" candidate_uuid: "65690ad1c05d4a8083522828aca777ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" is_pre_election: true
I20260925 10:32:56.845382 7762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3bf5a562034c82a9e2035d712eeb17" candidate_uuid: "65690ad1c05d4a8083522828aca777ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" is_pre_election: true
I20260925 10:32:56.845594 7622 raft_consensus.cc:2454] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65690ad1c05d4a8083522828aca777ba in term 0.
I20260925 10:32:56.845734 7762 raft_consensus.cc:2454] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65690ad1c05d4a8083522828aca777ba in term 0.
I20260925 10:32:56.846029 7410 leader_election.cc:302] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba74a37e8f444618ecb68f3c6a44c88, 65690ad1c05d4a8083522828aca777ba; no voters:
I20260925 10:32:56.846320 7922 raft_consensus.cc:2781] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:32:56.846460 7922 raft_consensus.cc:482] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:32:56.846566 7922 raft_consensus.cc:3037] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:56.848551 7922 raft_consensus.cc:504] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } } peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } }
I20260925 10:32:56.849058 7922 leader_election.cc:288] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [CANDIDATE]: Term 1 election: Requested vote from peers 2ba74a37e8f444618ecb68f3c6a44c88 (127.0.97.2:33031), 9ba24b6ac5a349b3b3a88387a76dba5c (127.0.97.3:37187)
I20260925 10:32:56.849556 7622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3bf5a562034c82a9e2035d712eeb17" candidate_uuid: "65690ad1c05d4a8083522828aca777ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ba74a37e8f444618ecb68f3c6a44c88"
I20260925 10:32:56.849556 7762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3bf5a562034c82a9e2035d712eeb17" candidate_uuid: "65690ad1c05d4a8083522828aca777ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c"
I20260925 10:32:56.849748 7622 raft_consensus.cc:3037] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:56.849749 7762 raft_consensus.cc:3037] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:56.852643 7622 raft_consensus.cc:2454] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65690ad1c05d4a8083522828aca777ba in term 1.
I20260925 10:32:56.852643 7762 raft_consensus.cc:2454] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65690ad1c05d4a8083522828aca777ba in term 1.
I20260925 10:32:56.853008 7410 leader_election.cc:302] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba74a37e8f444618ecb68f3c6a44c88, 65690ad1c05d4a8083522828aca777ba; no voters:
I20260925 10:32:56.853317 7922 raft_consensus.cc:2781] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:56.853775 7922 raft_consensus.cc:686] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [term 1 LEADER]: Becoming Leader. State: Replica: 65690ad1c05d4a8083522828aca777ba, State: Running, Role: LEADER
I20260925 10:32:56.854061 7922 consensus_queue.cc:237] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } } peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } }
I20260925 10:32:56.854729 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.854340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.1:46103)
0925 10:32:56.854488 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.854578 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 10:32:56.856788 7338 catalog_manager.cc:5495] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 65690ad1c05d4a8083522828aca777ba (127.0.97.1). New cstate: current_term: 1 leader_uuid: "65690ad1c05d4a8083522828aca777ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65690ad1c05d4a8083522828aca777ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33123 } health_report { overall_health: HEALTHY } } }
I20260925 10:32:56.914448 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.910024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.0.97.1:33123)
0925 10:32:56.910223 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:56.910231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:56.910251 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:32:56.910344 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:56.910349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:56.910419 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:56.910512 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:32:56.910849 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:56.911525 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:56.912818 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:56.913228 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:56.913357 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:56.913727 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:32:56.913820 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:32:56.913912 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:56.914055 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 10:32:56.914126 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:32:56.917388 7762 raft_consensus.cc:1261] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 1 FOLLOWER]: Refusing update from remote peer 65690ad1c05d4a8083522828aca777ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:56.917415 7622 raft_consensus.cc:1261] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 1 FOLLOWER]: Refusing update from remote peer 65690ad1c05d4a8083522828aca777ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:32:56.918036 7923 consensus_queue.cc:1035] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ba24b6ac5a349b3b3a88387a76dba5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:56.918385 7922 consensus_queue.cc:1035] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ba74a37e8f444618ecb68f3c6a44c88" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:32:56.921277 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.920771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.2:43119)
0925 10:32:56.920987 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.921079 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 10:32:56.921819 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.921217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.3:35161)
0925 10:32:56.921502 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.921607 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260925 10:32:56.927780 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.926829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.3:39855)
0925 10:32:56.927446 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.927566 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467}
I20260925 10:32:56.929190 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.928712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.1:51623)
0925 10:32:56.928923 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.929026 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 10:32:56.933197 7339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:56.933530 7339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:32:56.939090 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.938354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.2:48843)
0925 10:32:56.938711 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.938843 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260925 10:32:56.940920 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.940302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.97.1:51691)
0925 10:32:56.940577 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:56.940688 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260925 10:32:56.942579 7339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6: Sending DeleteTablet for 3 replicas of tablet 6b3bf5a562034c82a9e2035d712eeb17
I20260925 10:32:56.943928 7743 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3bf5a562034c82a9e2035d712eeb17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:56 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:34348
I20260925 10:32:56.944123 7602 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3bf5a562034c82a9e2035d712eeb17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:56 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:36144
I20260925 10:32:56.944751 7454 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3bf5a562034c82a9e2035d712eeb17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:32:56 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:42644
I20260925 10:32:56.945569 7937 tablet_replica.cc:330] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba: stopping tablet replica
I20260925 10:32:56.945590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 7312
I20260925 10:32:56.945746 7937 raft_consensus.cc:2229] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:32:56.946121 7937 raft_consensus.cc:2258] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:56.947724 7936 tablet_replica.cc:330] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88: stopping tablet replica
I20260925 10:32:56.951100 7937 ts_tablet_manager.cc:1905] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:56.952792 7936 raft_consensus.cc:2229] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:56.953610 7936 raft_consensus.cc:2258] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:56.954839 7935 tablet_replica.cc:330] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c: stopping tablet replica
I20260925 10:32:56.955094 7935 raft_consensus.cc:2229] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:32:56.955430 7935 raft_consensus.cc:2258] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:32:56.956619 7935 ts_tablet_manager.cc:1905] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:56.956933 7936 ts_tablet_manager.cc:1905] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:32:56.959283 7937 ts_tablet_manager.cc:1918] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:32:56.959407 7937 log.cc:1196] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/6b3bf5a562034c82a9e2035d712eeb17
I20260925 10:32:56.959869 7937 ts_tablet_manager.cc:1939] T 6b3bf5a562034c82a9e2035d712eeb17 P 65690ad1c05d4a8083522828aca777ba: Deleting consensus metadata
I20260925 10:32:56.960922 7326 catalog_manager.cc:4841] TS 65690ad1c05d4a8083522828aca777ba (127.0.97.1:33123): tablet 6b3bf5a562034c82a9e2035d712eeb17 (table test-table [id=642c8a66edaf4e328acafeee72de9c37]) successfully deleted
I20260925 10:32:56.962281 7935 ts_tablet_manager.cc:1918] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:32:56.962478 7935 log.cc:1196] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/6b3bf5a562034c82a9e2035d712eeb17
I20260925 10:32:56.962903 7935 ts_tablet_manager.cc:1939] T 6b3bf5a562034c82a9e2035d712eeb17 P 9ba24b6ac5a349b3b3a88387a76dba5c: Deleting consensus metadata
I20260925 10:32:56.963775 7324 catalog_manager.cc:4841] TS 9ba24b6ac5a349b3b3a88387a76dba5c (127.0.97.3:37187): tablet 6b3bf5a562034c82a9e2035d712eeb17 (table test-table [id=642c8a66edaf4e328acafeee72de9c37]) successfully deleted
I20260925 10:32:56.964090 7936 ts_tablet_manager.cc:1918] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:32:56.964193 7936 log.cc:1196] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/6b3bf5a562034c82a9e2035d712eeb17
I20260925 10:32:56.964565 7936 ts_tablet_manager.cc:1939] T 6b3bf5a562034c82a9e2035d712eeb17 P 2ba74a37e8f444618ecb68f3c6a44c88: Deleting consensus metadata
I20260925 10:32:56.965622 7326 catalog_manager.cc:4841] TS 2ba74a37e8f444618ecb68f3c6a44c88 (127.0.97.2:33031): tablet 6b3bf5a562034c82a9e2035d712eeb17 (table test-table [id=642c8a66edaf4e328acafeee72de9c37]) successfully deleted
I20260925 10:32:56.972213 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:56.946762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.0.97.1:33123)
0925 10:32:56.946918 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:56.946927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:56.956050 (+ 9123us) server_negotiation.cc:407] Connection header received
0925 10:32:56.956313 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:56.956319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:56.956397 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:56.956504 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:56.960485 (+ 3981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:56.961560 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:56.964068 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:56.964645 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:56.964748 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:56.967509 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:56.967521 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:56.967536 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:56.967564 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:56.970081 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:56.970689 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:56.970695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:56.970697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:56.970816 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:56.971106 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:56.971109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:56.971111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:56.971446 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:56.971657 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:56.971847 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 10:32:56.971991 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:32:56.972877 7514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:57.065651 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:57.064957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:49004)
0925 10:32:57.065392 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:57.065476 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":244,"threads_started":1}
I20260925 10:32:57.066900 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:57.066600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46413 (local address 127.0.0.1:49008)
0925 10:32:57.066732 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:57.066800 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 10:32:57.093305 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 7312
I20260925 10:32:57.115063 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 7531
I20260925 10:32:57.127377 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:57.115824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.0.97.2:33031)
0925 10:32:57.116269 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:57.116275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:57.116293 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:32:57.116361 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:57.116366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:57.116443 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:57.116530 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:57.117693 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:57.118447 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:57.119632 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:57.120018 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:57.120105 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:57.122553 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:57.122565 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:57.122580 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:57.122619 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:57.125059 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:57.125686 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:57.125693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:57.125695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:57.125808 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:57.126090 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:57.126093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:57.126095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:57.126491 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:57.126724 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:57.126896 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 10:32:57.127036 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":205,"threads_started":1}
I20260925 10:32:57.127997 7662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:57.218237 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 7531
I20260925 10:32:57.234618 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 7680
I20260925 10:32:57.247397 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:57.235328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.0.97.3:37187)
0925 10:32:57.235785 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:57.235789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:57.235804 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:32:57.235850 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:57.235854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:57.235914 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:57.236023 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:57.237228 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:57.237937 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:57.239171 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:57.239558 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:57.239671 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:57.242194 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:57.242208 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:57.242224 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:57.242254 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:57.244851 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:57.245452 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:57.245464 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:57.245467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:57.245591 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:57.245948 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:57.245955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:57.245958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:57.246280 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:57.246647 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:57.246878 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 10:32:57.247086 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260925 10:32:57.248008 7803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:57.338053 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 7680
I20260925 10:32:57.354425 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 7007
I20260925 10:32:57.355472 7173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:57.467989 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 7007
W20260925 10:32:57.487897 7326 connection.cc:535] client connection to 127.0.97.61:43321 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:32:57.488274 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 7107
I20260925 10:32:57.488986 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:57.488554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43321 (local address 127.0.0.1:60540)
0925 10:32:57.488740 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:57.488832 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 10:32:57.489168 7326 consensus_peers.cc:487] T 00000000000000000000000000000000 P e36414eb5d1d42b6a5461ee76d162ca6 -> Peer ca8ca5ced019409bacfa9b2e62f58a58 (127.0.97.61:43321): Couldn't send request to peer ca8ca5ced019409bacfa9b2e62f58a58. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 10:32:57.489270 7369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:32:57.608049 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 7107
2026-09-25T10:32:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11283 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:32:57 dist-test-slave-dxdd krb5kdc[7950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:32:57 dist-test-slave-dxdd krb5kdc[7950](info): set up 2 sockets
Sep 25 10:32:57 dist-test-slave-dxdd krb5kdc[7950](info): commencing operation
krb5kdc: starting...
W20260925 10:32:58.691118 388 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:32:58 dist-test-slave-dxdd krb5kdc[7950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:32:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:32:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260925 10:32:58.891312 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39929
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:59.201375 7963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:59.201654 7963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:59.201804 7963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:59.208875 7963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:32:59.208967 7963 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:32:59.209059 7963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:59.209161 7963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:59.209236 7963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:32:59.209334 7963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:32:59.218907 7963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39929
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:59.220121 7963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:59.221369 7963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:59.233088 7969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:59.233124 7968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:59.233043 7971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:59.233327 7963 server_base.cc:1034] running on GCE node
I20260925 10:32:59.234368 7963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:59.235574 7963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:59.236809 7963 hybrid_clock.cc:648] HybridClock initialized: now 1790332379236750 us; error 33 us; skew 500 ppm
I20260925 10:32:59.237167 7963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:59 dist-test-slave-dxdd krb5kdc[7950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332379, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:59.240278 7963 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260925 10:32:59.241554 7963 webserver.cc:460] Webserver started at http://127.0.97.62:35719/ using document root <none> and password file <none>
I20260925 10:32:59.242236 7963 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:59.242383 7963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:59.242705 7963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:59.245468 7963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "773b290855614644aebd3d2521477b89"
format_stamp: "Formatted at 2026-09-25 10:32:59 on dist-test-slave-dxdd"
server_key: "ead09feefc41e0403537febbbded69f0"
server_key_iv: "af0724c97dde1646473da01f479828bb"
server_key_version: "encryptionkey@0"
I20260925 10:32:59.246263 7963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "773b290855614644aebd3d2521477b89"
format_stamp: "Formatted at 2026-09-25 10:32:59 on dist-test-slave-dxdd"
server_key: "ead09feefc41e0403537febbbded69f0"
server_key_iv: "af0724c97dde1646473da01f479828bb"
server_key_version: "encryptionkey@0"
I20260925 10:32:59.251479 7963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:32:59.254729 7978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:59.255954 7963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:32:59.256129 7963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "773b290855614644aebd3d2521477b89"
format_stamp: "Formatted at 2026-09-25 10:32:59 on dist-test-slave-dxdd"
server_key: "ead09feefc41e0403537febbbded69f0"
server_key_iv: "af0724c97dde1646473da01f479828bb"
server_key_version: "encryptionkey@0"
I20260925 10:32:59.256330 7963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:59.281205 7963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:59.285341 7963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:59.285632 7963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:59.305714 7963 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39929
I20260925 10:32:59.305733 8039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39929 every 8 connection(s)
I20260925 10:32:59.306988 7963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:32:59.309049 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 7963
I20260925 10:32:59.309239 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:32:59.309855 388 external_mini_cluster.cc:1428] Setting key c0fab5c4d66bca6a1f1dd49197c743da
I20260925 10:32:59.310557 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:32:59.322127 8040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89: Bootstrap starting.
Sep 25 10:32:59 dist-test-slave-dxdd krb5kdc[7950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260925 10:32:59.325147 8040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89: Neither blocks nor log segments found. Creating new log.
I20260925 10:32:59.326159 8040 log.cc:824] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89: Log is configured to *not* fsync() on all Append() calls
I20260925 10:32:59.328657 8040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89: No bootstrap required, opened a new log
I20260925 10:32:59.330380 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:59.311720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48980 (local address 127.0.97.62:39929)
0925 10:32:59.312403 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:59.312414 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:59.312451 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:32:59.313900 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:59.313918 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:59.314271 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:59.314699 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:59.316061 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:59.317368 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:59.318760 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:59.319813 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:59.320009 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:59.323635 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:59.323644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:59.323656 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:59.323691 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:59.326698 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:59.327348 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:59.327360 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:59.327371 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:59.327520 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:59.327879 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:59.327885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:59.327888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:59.328320 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:59.328611 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:59.329004 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 10:32:59.329424 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":308,"threads_started":1}
I20260925 10:32:59.335847 8040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773b290855614644aebd3d2521477b89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39929 } }
I20260925 10:32:59.336153 8040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:32:59.336263 8040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773b290855614644aebd3d2521477b89, State: Initialized, Role: FOLLOWER
I20260925 10:32:59.336727 8040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773b290855614644aebd3d2521477b89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39929 } }
I20260925 10:32:59.336939 8040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:32:59.337064 8040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:32:59.337224 8040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:32:59.339500 8040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773b290855614644aebd3d2521477b89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39929 } }
I20260925 10:32:59.339918 8040 leader_election.cc:302] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 773b290855614644aebd3d2521477b89; no voters:
I20260925 10:32:59.340359 8040 leader_election.cc:288] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:32:59.340458 8045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:32:59.341112 8045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [term 1 LEADER]: Becoming Leader. State: Replica: 773b290855614644aebd3d2521477b89, State: Running, Role: LEADER
I20260925 10:32:59.341638 8045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773b290855614644aebd3d2521477b89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39929 } }
I20260925 10:32:59.342177 8040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:32:59.344794 8047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 773b290855614644aebd3d2521477b89. Latest consensus state: current_term: 1 leader_uuid: "773b290855614644aebd3d2521477b89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773b290855614644aebd3d2521477b89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39929 } } }
I20260925 10:32:59.345227 8047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:59.346585 8046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "773b290855614644aebd3d2521477b89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773b290855614644aebd3d2521477b89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39929 } } }
I20260925 10:32:59.346861 8046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89 [sys.catalog]: This master's current role is: LEADER
I20260925 10:32:59.347424 8052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:32:59.352147 8052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:32:59.359618 8052 catalog_manager.cc:1347] Generated new cluster ID: feace45e537646bd8a6701b02849d1b7
I20260925 10:32:59.359714 8052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:32:59.373574 8052 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:32:59.374827 8052 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:32:59.388811 8052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89: Generated new TSK 0
I20260925 10:32:59.389554 8052 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260925 10:32:59.447520 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39929
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:32:59.760550 8066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:32:59.760812 8066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:32:59.760982 8066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:32:59.768141 8066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:32:59.768272 8066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:32:59.768421 8066 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:32:59.768544 8066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:32:59.778483 8066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:32:59.779809 8066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:32:59.781208 8066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:32:59.795691 8072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:59.795749 8074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:32:59.795775 8071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:32:59.799652 8066 server_base.cc:1034] running on GCE node
I20260925 10:32:59.800315 8066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:32:59.801272 8066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:32:59.802500 8066 hybrid_clock.cc:648] HybridClock initialized: now 1790332379802443 us; error 40 us; skew 500 ppm
I20260925 10:32:59.802817 8066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:32:59 dist-test-slave-dxdd krb5kdc[7950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332379, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:32:59.805786 8066 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260925 10:32:59.807273 8066 webserver.cc:460] Webserver started at http://127.0.97.1:43243/ using document root <none> and password file <none>
I20260925 10:32:59.807972 8066 fs_manager.cc:359] Metadata directory not provided
I20260925 10:32:59.808096 8066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:32:59.808410 8066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:32:59.811277 8066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "e9c557346c1141d58968601009179324"
format_stamp: "Formatted at 2026-09-25 10:32:59 on dist-test-slave-dxdd"
server_key: "043c0a819d9d8734894f00fddcc07573"
server_key_iv: "e6f6a172733309793d4da5570cf9863f"
server_key_version: "encryptionkey@0"
I20260925 10:32:59.812052 8066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e9c557346c1141d58968601009179324"
format_stamp: "Formatted at 2026-09-25 10:32:59 on dist-test-slave-dxdd"
server_key: "043c0a819d9d8734894f00fddcc07573"
server_key_iv: "e6f6a172733309793d4da5570cf9863f"
server_key_version: "encryptionkey@0"
I20260925 10:32:59.816919 8066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:32:59.820190 8081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:59.821471 8066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:32:59.821656 8066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "e9c557346c1141d58968601009179324"
format_stamp: "Formatted at 2026-09-25 10:32:59 on dist-test-slave-dxdd"
server_key: "043c0a819d9d8734894f00fddcc07573"
server_key_iv: "e6f6a172733309793d4da5570cf9863f"
server_key_version: "encryptionkey@0"
I20260925 10:32:59.821877 8066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:32:59.845376 8066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:32:59.849478 8066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:32:59.849756 8066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:32:59.852320 8066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:32:59.852421 8066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:32:59.852558 8066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:32:59.852634 8066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:32:59 dist-test-slave-dxdd krb5kdc[7950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332379, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260925 10:32:59.874058 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:59.852937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42333 (local address 127.0.97.62:39929)
0925 10:32:59.853558 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:59.853564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:59.854656 (+ 1092us) server_negotiation.cc:407] Connection header received
0925 10:32:59.856972 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:59.856977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:59.857040 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:59.857137 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:59.859642 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:59.860920 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:59.863266 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:59.864038 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:59.864188 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:59.867636 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:59.867646 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:59.867654 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:59.867700 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:59.871006 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:59.871786 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:59.871794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:59.871801 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:59.871965 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:59.872448 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:59.872453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:59.872456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:59.872721 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:59.872869 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:59.873492 (+ 623us) server_negotiation.cc:299] Negotiation successful
0925 10:32:59.873696 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":393,"threads_started":1}
I20260925 10:32:59.875239 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:59.853628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39929 (local address 127.0.97.1:42333)
0925 10:32:59.854493 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:59.854544 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:59.856609 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:59.857643 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:59.857653 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:59.858513 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:59.859398 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:59.859417 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:59.861084 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:59.861092 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:59.863100 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:59.863119 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:59.864292 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:59.864305 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:59.864513 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:59.865140 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:59.865160 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:59.867414 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:59.871136 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:59.871144 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:59.871162 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:59.871636 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:59.872084 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:59.872091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:59.872100 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:59.872312 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:59.872840 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:59.872847 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:59.873158 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 10:32:59.873398 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 10:32:59.873881 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":353,"threads_started":1}
I20260925 10:32:59.898061 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:59.880558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39929 (local address 127.0.97.1:42531)
0925 10:32:59.882908 (+ 2350us) negotiation.cc:107] Waiting for socket to connect
0925 10:32:59.882932 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:32:59.883089 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:32:59.883399 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:32:59.883407 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:32:59.884144 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:32:59.884586 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:59.884601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:59.885520 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:59.885528 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:59.887765 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:32:59.887783 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:32:59.889297 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:32:59.889303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:32:59.889579 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:59.890393 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:32:59.890418 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:32:59.892027 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:32:59.895551 (+ 3524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:59.895555 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:59.895557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:59.895986 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:59.896474 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:32:59.896478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:32:59.896480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:32:59.896589 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:32:59.897268 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:32:59.897271 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:32:59.897414 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 10:32:59.897531 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 10:32:59.897745 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2167,"thread_start_us":1971,"threads_started":1}
I20260925 10:32:59.898489 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:32:59.880610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42531 (local address 127.0.97.62:39929)
0925 10:32:59.880779 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 10:32:59.880787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:32:59.883024 (+ 2237us) server_negotiation.cc:407] Connection header received
0925 10:32:59.883215 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:32:59.883219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:32:59.883272 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:32:59.883342 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:32:59.884734 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:59.885402 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:59.887975 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:32:59.888778 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:32:59.888905 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:32:59.892225 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:32:59.892232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:32:59.892239 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:32:59.892279 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:32:59.895374 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:59.896133 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:59.896142 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:59.896148 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:59.896346 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:32:59.896701 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:32:59.896709 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:32:59.896714 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:32:59.897145 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:32:59.897331 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 10:32:59.897687 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 10:32:59.897969 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:32:59.907982 8066 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39373
I20260925 10:32:59.908023 8201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39373 every 8 connection(s)
I20260925 10:32:59.909123 8066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:32:59.911046 8202 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39929
I20260925 10:32:59.911263 8202 heartbeater.cc:464] Registering TS with master...
I20260925 10:32:59.911840 8202 heartbeater.cc:511] Master 127.0.97.62:39929 requested a full tablet report, sending...
I20260925 10:32:59.913590 7995 ts_manager.cc:194] Registered new tserver with Master: e9c557346c1141d58968601009179324 (127.0.97.1:39373)
I20260925 10:32:59.915006 7995 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42333
I20260925 10:32:59.917742 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 8066
I20260925 10:32:59.917928 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:32:59.918591 388 external_mini_cluster.cc:1428] Setting key 2e1620abb7b7ad1ea3652ad7f6ea5f59
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260925 10:32:59.961912 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39929
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:33:00.269577 8206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:00.269825 8206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:00.269917 8206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:00.276942 8206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:00.277040 8206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:00.277195 8206 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:33:00.277313 8206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:33:00.286900 8206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:00.288142 8206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:00.289737 8206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:00.302677 8212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:00.302742 8211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:00.302774 8214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:00.303733 8206 server_base.cc:1034] running on GCE node
I20260925 10:33:00.304420 8206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:00.305413 8206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:00.306630 8206 hybrid_clock.cc:648] HybridClock initialized: now 1790332380306569 us; error 39 us; skew 500 ppm
I20260925 10:33:00.306931 8206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:33:00 dist-test-slave-dxdd krb5kdc[7950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332380, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:33:00.310123 8206 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260925 10:33:00.311601 8206 webserver.cc:460] Webserver started at http://127.0.97.2:40637/ using document root <none> and password file <none>
I20260925 10:33:00.312286 8206 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:00.312409 8206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:00.312723 8206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:00.315519 8206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "5a514a91eeaa47428ddd63e5facdd3ae"
format_stamp: "Formatted at 2026-09-25 10:33:00 on dist-test-slave-dxdd"
server_key: "6cdc3017d46a7e4ef4ce9bbcf921b5b1"
server_key_iv: "17aeb276dd766793a70f216579b76daa"
server_key_version: "encryptionkey@0"
I20260925 10:33:00.316239 8206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5a514a91eeaa47428ddd63e5facdd3ae"
format_stamp: "Formatted at 2026-09-25 10:33:00 on dist-test-slave-dxdd"
server_key: "6cdc3017d46a7e4ef4ce9bbcf921b5b1"
server_key_iv: "17aeb276dd766793a70f216579b76daa"
server_key_version: "encryptionkey@0"
I20260925 10:33:00.320988 8206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:33:00.324177 8221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:00.325469 8206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:33:00.325623 8206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "5a514a91eeaa47428ddd63e5facdd3ae"
format_stamp: "Formatted at 2026-09-25 10:33:00 on dist-test-slave-dxdd"
server_key: "6cdc3017d46a7e4ef4ce9bbcf921b5b1"
server_key_iv: "17aeb276dd766793a70f216579b76daa"
server_key_version: "encryptionkey@0"
I20260925 10:33:00.325893 8206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:00.352381 8206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:00.356258 8206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:00.356554 8206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:00.359263 8206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:00.359385 8206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:00.359561 8206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:00.359673 8206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:33:00 dist-test-slave-dxdd krb5kdc[7950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332380, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260925 10:33:00.382941 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:00.359575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57691 (local address 127.0.97.62:39929)
0925 10:33:00.359726 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:00.359731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:00.361202 (+ 1471us) server_negotiation.cc:407] Connection header received
0925 10:33:00.363409 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:00.363414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:00.363527 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:00.363641 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:00.366358 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.367412 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.369914 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.370537 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.370705 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:00.375803 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:00.375812 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:00.375818 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:00.375866 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:00.379247 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:00.380333 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:00.380341 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:00.380347 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:00.380512 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:00.381170 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:00.381178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:00.381182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:00.381564 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:00.381724 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:00.382378 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 10:33:00.382594 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:33:00.384385 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:00.360279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39929 (local address 127.0.97.2:57691)
0925 10:33:00.361065 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:00.361101 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:00.363105 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:00.363972 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:00.363982 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:00.364908 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:00.366121 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:00.366157 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.367577 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.367586 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:00.369746 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:00.369761 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.371048 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.371053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:00.371344 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:00.372140 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:00.372180 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:00.375580 (+ 3400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:00.379400 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:00.379413 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:00.379433 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:00.380199 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:00.380657 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:00.380665 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:00.380671 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:00.381045 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:00.381713 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:00.381720 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:00.382040 (+ 320us) client_negotiation.cc:769] Sending connection context
0925 10:33:00.382288 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 10:33:00.382847 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":341,"threads_started":1}
I20260925 10:33:00.402585 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:00.390389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33103 (local address 127.0.97.62:39929)
0925 10:33:00.390538 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:00.390542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:00.391215 (+ 673us) server_negotiation.cc:407] Connection header received
0925 10:33:00.391376 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:00.391385 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:00.391437 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:00.391526 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:00.392355 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.393027 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.394335 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.394938 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.395068 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:00.397449 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:00.397455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:00.397465 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:00.397493 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:00.400200 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:00.400789 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:00.400796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:00.400799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:00.400920 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:00.401261 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:00.401267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:00.401270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:00.401546 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:00.401711 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:00.402062 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 10:33:00.402262 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:33:00.403469 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:00.390484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39929 (local address 127.0.97.2:33103)
0925 10:33:00.391123 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:00.391141 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:00.391273 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:00.391545 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:00.391550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:00.391956 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:00.392208 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:00.392216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.393149 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.393155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:00.394168 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:00.394179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.395127 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.395136 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:00.395295 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:00.396170 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:00.396203 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:00.397264 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:00.400314 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:00.400320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:00.400324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:00.400679 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:00.401027 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:00.401032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:00.401036 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:00.401153 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:00.401676 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:00.401680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:00.401846 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 10:33:00.401978 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 10:33:00.402262 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":389,"threads_started":1}
I20260925 10:33:00.410818 8206 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44725
I20260925 10:33:00.410859 8340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44725 every 8 connection(s)
I20260925 10:33:00.411949 8206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:33:00.413781 8341 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39929
I20260925 10:33:00.413985 8341 heartbeater.cc:464] Registering TS with master...
I20260925 10:33:00.414614 8341 heartbeater.cc:511] Master 127.0.97.62:39929 requested a full tablet report, sending...
I20260925 10:33:00.415764 7995 ts_manager.cc:194] Registered new tserver with Master: 5a514a91eeaa47428ddd63e5facdd3ae (127.0.97.2:44725)
I20260925 10:33:00.416499 7995 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57691
I20260925 10:33:00.420621 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 8206
I20260925 10:33:00.420746 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:33:00.421149 388 external_mini_cluster.cc:1428] Setting key 46f61a3dfe405464dee4b196d30b9f9b
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260925 10:33:00.463717 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39929
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:33:00.774050 8345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:00.774355 8345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:00.774542 8345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:00.781829 8345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:00.782030 8345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:00.782189 8345 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:33:00.782305 8345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:33:00.792320 8345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:00.793922 8345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:00.795603 8345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:00.808804 8350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:00.808772 8353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:00.808729 8351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:00.809880 8345 server_base.cc:1034] running on GCE node
I20260925 10:33:00.810524 8345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:00.811378 8345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:00.812609 8345 hybrid_clock.cc:648] HybridClock initialized: now 1790332380812538 us; error 55 us; skew 500 ppm
I20260925 10:33:00.812909 8345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:33:00 dist-test-slave-dxdd krb5kdc[7950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332380, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:33:00.816030 8345 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260925 10:33:00.817324 8345 webserver.cc:460] Webserver started at http://127.0.97.3:33205/ using document root <none> and password file <none>
I20260925 10:33:00.817988 8345 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:00.818109 8345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:00.818581 8345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:00.821311 8345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "133464d0ed444863883f61a804eafde6"
format_stamp: "Formatted at 2026-09-25 10:33:00 on dist-test-slave-dxdd"
server_key: "5b4b92f5b03396cac290fa11374f342d"
server_key_iv: "3706d85115eda55942800cfd45a6e560"
server_key_version: "encryptionkey@0"
I20260925 10:33:00.822033 8345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "133464d0ed444863883f61a804eafde6"
format_stamp: "Formatted at 2026-09-25 10:33:00 on dist-test-slave-dxdd"
server_key: "5b4b92f5b03396cac290fa11374f342d"
server_key_iv: "3706d85115eda55942800cfd45a6e560"
server_key_version: "encryptionkey@0"
I20260925 10:33:00.826956 8345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 10:33:00.830263 8360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:00.831555 8345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:33:00.831732 8345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "133464d0ed444863883f61a804eafde6"
format_stamp: "Formatted at 2026-09-25 10:33:00 on dist-test-slave-dxdd"
server_key: "5b4b92f5b03396cac290fa11374f342d"
server_key_iv: "3706d85115eda55942800cfd45a6e560"
server_key_version: "encryptionkey@0"
I20260925 10:33:00.831939 8345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:00.846202 8345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:00.850672 8345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:00.850944 8345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:00.853560 8345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:00.853672 8345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:00.853854 8345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:00.853968 8345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:33:00 dist-test-slave-dxdd krb5kdc[7950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332380, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260925 10:33:00.875018 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:00.853908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60557 (local address 127.0.97.62:39929)
0925 10:33:00.854102 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:00.854106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:00.855394 (+ 1288us) server_negotiation.cc:407] Connection header received
0925 10:33:00.857454 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:00.857462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:00.857563 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:00.857681 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:00.859950 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.861323 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.863839 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.864235 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.864366 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:00.868191 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:00.868196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:00.868199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:00.868228 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:00.871172 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:00.872313 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:00.872319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:00.872323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:00.872447 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:00.873030 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:00.873036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:00.873042 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:00.873364 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:00.873545 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:00.874449 (+ 904us) server_negotiation.cc:299] Negotiation successful
0925 10:33:00.874665 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:33:00.875986 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:00.854538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39929 (local address 127.0.97.3:60557)
0925 10:33:00.855243 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:00.855285 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:00.857083 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:00.857915 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:00.857931 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:00.858765 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:00.859646 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:00.859668 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.861503 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.861511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:00.863684 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:00.863694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.864462 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.864469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:00.864667 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:00.865458 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:00.865478 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:00.867982 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:00.871347 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:00.871363 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:00.871382 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:00.872169 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:00.872562 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:00.872570 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:00.872577 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:00.872886 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:00.873631 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:00.873641 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:00.874018 (+ 377us) client_negotiation.cc:769] Sending connection context
0925 10:33:00.874273 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 10:33:00.874810 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":292,"threads_started":1}
I20260925 10:33:00.905028 8345 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40741
I20260925 10:33:00.905051 8479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40741 every 8 connection(s)
I20260925 10:33:00.906013 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:00.889845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39929 (local address 127.0.97.3:51271)
0925 10:33:00.890913 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:00.890933 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:00.891115 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:00.891546 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:00.891552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:00.891964 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:00.892247 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:00.892254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.893807 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.893815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:00.895783 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:00.895802 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.896789 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.896794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:00.896907 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:00.897893 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:00.897927 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:00.899231 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:00.903372 (+ 4141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:00.903376 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:00.903379 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:00.903893 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:00.904332 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:00.904339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:00.904345 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:00.904518 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:00.905139 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:00.905144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:00.905344 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 10:33:00.905477 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 10:33:00.905715 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":597,"threads_started":1}
I20260925 10:33:00.906181 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:00.889803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51271 (local address 127.0.97.62:39929)
0925 10:33:00.890011 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:00.890020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:00.891051 (+ 1031us) server_negotiation.cc:407] Connection header received
0925 10:33:00.891265 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:00.891273 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:00.891376 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:00.891500 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:00.892380 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.893639 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.895954 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.896621 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.896757 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:00.899428 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:00.899437 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:00.899445 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:00.899491 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:00.903262 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:00.904044 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:00.904054 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:00.904059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:00.904223 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:00.904644 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:00.904649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:00.904652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:00.905017 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:00.905170 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:00.905626 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 10:33:00.905832 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:33:00.906600 8345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:33:00.908689 8480 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39929
I20260925 10:33:00.908927 8480 heartbeater.cc:464] Registering TS with master...
I20260925 10:33:00.909518 8480 heartbeater.cc:511] Master 127.0.97.62:39929 requested a full tablet report, sending...
I20260925 10:33:00.910808 7995 ts_manager.cc:194] Registered new tserver with Master: 133464d0ed444863883f61a804eafde6 (127.0.97.3:40741)
I20260925 10:33:00.911442 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 8345
I20260925 10:33:00.911597 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:33:00.911921 7995 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60557
I20260925 10:33:00.911994 388 external_mini_cluster.cc:1428] Setting key 7161b8df9a19bce0e8bad03b1d651e07
I20260925 10:33:00.914947 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:33:00.917344 8202 heartbeater.cc:503] Master 127.0.97.62:39929 was elected leader, sending a full tablet report...
Sep 25 10:33:00 dist-test-slave-dxdd krb5kdc[7950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332380, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:33:00 dist-test-slave-dxdd krb5kdc[7950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:00 dist-test-slave-dxdd krb5kdc[7950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:33:00.960979 8090 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:00.950933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.0.97.62:39929)
0925 10:33:00.951164 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:00.951172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:00.951388 (+ 216us) server_negotiation.cc:407] Connection header received
0925 10:33:00.951594 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:00.951603 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:00.951663 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:00.951764 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:00.953031 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.953722 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.955048 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.955459 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.955585 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:00.960682 (+ 5097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48994: BlockingRecv error: recv got EOF from 127.0.0.1:48994 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 25 10:33:00 dist-test-slave-dxdd krb5kdc[7950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332380, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260925 10:33:00.975404 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:00.963881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.0.97.62:39929)
0925 10:33:00.964047 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:00.964051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:00.964309 (+ 258us) server_negotiation.cc:407] Connection header received
0925 10:33:00.964407 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:00.964416 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:00.964486 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:00.964567 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:00.965594 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.966281 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.967479 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:00.967860 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:00.967950 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:00.971219 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:00.971224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:00.971227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:00.971254 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:00.973480 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:00.973993 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:00.973996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:00.973999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:00.974113 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:00.974475 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:00.974479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:00.974480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:00.974702 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:00.974894 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:00.975048 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 10:33:00.975191 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:33:00.978915 7995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:33:00.985436 7995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:33:01.017982 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:00.996869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.0.97.2:44725)
0925 10:33:01.007579 (+ 10710us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.007592 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.009576 (+ 1984us) server_negotiation.cc:407] Connection header received
0925 10:33:01.009679 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.009688 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.009940 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.010125 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:33:01.011260 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.012764 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.015588 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.016666 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.016787 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.016992 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.017529 (+ 537us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.017682 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10502,"thread_start_us":10724,"threads_started":1}
I20260925 10:33:01.018107 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:00.996257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44725 (local address 127.0.0.1:42344)
0925 10:33:01.009301 (+ 13044us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:01.009319 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:01.009449 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:01.010393 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:01.010400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:01.010448 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:33:01.011024 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.011037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.012938 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.012945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.015439 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.015452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.017141 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.017147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.017306 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.017337 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 10:33:01.017458 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 10:33:01.017660 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12911,"mutex_wait_us":167,"spinlock_wait_cycles":1280,"thread_start_us":12795,"threads_started":1}
I20260925 10:33:01.019858 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:00.996063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40741 (local address 127.0.0.1:48776)
0925 10:33:01.008454 (+ 12391us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:01.008501 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:01.008713 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:01.010393 (+ 1680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:01.010400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:01.010455 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:33:01.011121 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.011134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.013018 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.013023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.015452 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.015462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.019155 (+ 3693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.019159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.019260 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.019281 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 10:33:01.019376 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 10:33:01.019498 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12057,"mutex_wait_us":90,"spinlock_wait_cycles":26880,"thread_start_us":11878,"threads_started":1}
I20260925 10:33:01.019886 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:00.996383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.0.97.3:40741)
0925 10:33:01.009693 (+ 13310us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.009701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.009726 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:33:01.009798 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.009804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.009986 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.010138 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:33:01.011318 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.012889 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.016296 (+ 3407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.017700 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.017823 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.018023 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.019431 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.019576 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13148,"thread_start_us":12975,"threads_started":1}
I20260925 10:33:01.020944 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.009773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39373 (local address 127.0.0.1:40166)
0925 10:33:01.013467 (+ 3694us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:01.013484 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:01.013603 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:01.014265 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:01.014270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:01.014294 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:33:01.014813 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.014824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.016205 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.016209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.018268 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.018280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.020275 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.020283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.020435 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.020463 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 10:33:01.020534 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 10:33:01.020648 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3556,"thread_start_us":3371,"threads_started":1}
I20260925 10:33:01.021023 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.009250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40166 (local address 127.0.97.1:39373)
0925 10:33:01.013011 (+ 3761us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.013019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.013639 (+ 620us) server_negotiation.cc:407] Connection header received
0925 10:33:01.013859 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.013867 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.014082 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.014261 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:33:01.014999 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.016081 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.018424 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.019508 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.020315 (+ 807us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.020520 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.020597 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.020744 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3598,"thread_start_us":3463,"threads_started":1}
I20260925 10:33:01.021826 8271 tablet_service.cc:1467] Processing CreateTablet for tablet dfa7156f0a3049ad91b1925aba8ff8ed (DEFAULT_TABLE table=test-table [id=9ba55f70c6df4bd89adac43b1e2cebe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:33:01.023025 8271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa7156f0a3049ad91b1925aba8ff8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:01.024375 8132 tablet_service.cc:1467] Processing CreateTablet for tablet dfa7156f0a3049ad91b1925aba8ff8ed (DEFAULT_TABLE table=test-table [id=9ba55f70c6df4bd89adac43b1e2cebe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:33:01.024374 8410 tablet_service.cc:1467] Processing CreateTablet for tablet dfa7156f0a3049ad91b1925aba8ff8ed (DEFAULT_TABLE table=test-table [id=9ba55f70c6df4bd89adac43b1e2cebe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:33:01.025681 8132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa7156f0a3049ad91b1925aba8ff8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:01.025691 8410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa7156f0a3049ad91b1925aba8ff8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:01.033788 8504 tablet_bootstrap.cc:492] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae: Bootstrap starting.
I20260925 10:33:01.037664 8504 tablet_bootstrap.cc:654] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:01.038708 8504 log.cc:824] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae: Log is configured to *not* fsync() on all Append() calls
I20260925 10:33:01.039705 8505 tablet_bootstrap.cc:492] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324: Bootstrap starting.
I20260925 10:33:01.041345 8506 tablet_bootstrap.cc:492] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6: Bootstrap starting.
I20260925 10:33:01.043439 8504 tablet_bootstrap.cc:492] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae: No bootstrap required, opened a new log
I20260925 10:33:01.043759 8504 ts_tablet_manager.cc:1397] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 10:33:01.043885 8505 tablet_bootstrap.cc:654] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:01.044646 8506 tablet_bootstrap.cc:654] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:01.044809 8505 log.cc:824] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324: Log is configured to *not* fsync() on all Append() calls
I20260925 10:33:01.045748 8506 log.cc:824] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6: Log is configured to *not* fsync() on all Append() calls
I20260925 10:33:01.054332 8505 tablet_bootstrap.cc:492] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324: No bootstrap required, opened a new log
I20260925 10:33:01.054687 8505 ts_tablet_manager.cc:1397] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260925 10:33:01.056551 8504 raft_consensus.cc:348] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } }
I20260925 10:33:01.057031 8504 raft_consensus.cc:374] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:01.057139 8504 raft_consensus.cc:729] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a514a91eeaa47428ddd63e5facdd3ae, State: Initialized, Role: FOLLOWER
I20260925 10:33:01.057787 8504 consensus_queue.cc:260] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } }
I20260925 10:33:01.057986 8506 tablet_bootstrap.cc:492] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6: No bootstrap required, opened a new log
I20260925 10:33:01.058282 8506 ts_tablet_manager.cc:1397] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.007s
I20260925 10:33:01.058944 8341 heartbeater.cc:503] Master 127.0.97.62:39929 was elected leader, sending a full tablet report...
I20260925 10:33:01.059621 8504 ts_tablet_manager.cc:1428] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260925 10:33:01.063588 8505 raft_consensus.cc:348] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } }
I20260925 10:33:01.063970 8505 raft_consensus.cc:374] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:01.064074 8505 raft_consensus.cc:729] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c557346c1141d58968601009179324, State: Initialized, Role: FOLLOWER
I20260925 10:33:01.064692 8505 consensus_queue.cc:260] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } }
I20260925 10:33:01.066569 8505 ts_tablet_manager.cc:1428] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260925 10:33:01.068710 8506 raft_consensus.cc:348] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } }
I20260925 10:33:01.069072 8506 raft_consensus.cc:374] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:01.069156 8506 raft_consensus.cc:729] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133464d0ed444863883f61a804eafde6, State: Initialized, Role: FOLLOWER
I20260925 10:33:01.069665 8506 consensus_queue.cc:260] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } }
I20260925 10:33:01.070662 8480 heartbeater.cc:503] Master 127.0.97.62:39929 was elected leader, sending a full tablet report...
I20260925 10:33:01.071092 8506 ts_tablet_manager.cc:1428] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6: Time spent starting tablet: real 0.013s user 0.010s sys 0.001s
I20260925 10:33:01.103060 8510 raft_consensus.cc:482] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:33:01.103356 8510 raft_consensus.cc:504] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } }
I20260925 10:33:01.104951 8510 leader_election.cc:288] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 133464d0ed444863883f61a804eafde6 (127.0.97.3:40741), e9c557346c1141d58968601009179324 (127.0.97.1:39373)
I20260925 10:33:01.111629 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.106121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33645 (local address 127.0.97.1:39373)
0925 10:33:01.106306 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.106333 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.106529 (+ 196us) server_negotiation.cc:407] Connection header received
0925 10:33:01.106665 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.106672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.106762 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.106851 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:33:01.107438 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.108144 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.109875 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.110671 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.110789 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.110914 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.111231 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.111368 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:33:01.111660 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.105567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42199 (local address 127.0.97.3:40741)
0925 10:33:01.105776 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.105783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.105912 (+ 129us) server_negotiation.cc:407] Connection header received
0925 10:33:01.106244 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.106249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.106345 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.106405 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:33:01.107206 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.107994 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.109852 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.110703 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.110849 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.110949 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.111295 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.111416 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:33:01.111775 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.105864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39373 (local address 127.0.97.2:33645)
0925 10:33:01.106429 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:01.106445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:01.106568 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:01.106877 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:01.106880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:01.106899 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:33:01.107279 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.107290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.108272 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.108276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.109712 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.109723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.110892 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.110898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.111085 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.111109 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 10:33:01.111182 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 10:33:01.111337 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":2,"thread_start_us":462,"threads_started":1}
I20260925 10:33:01.112039 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.105187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40741 (local address 127.0.97.2:42199)
0925 10:33:01.105786 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:01.105807 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:01.106050 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:01.106612 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:01.106616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:01.106642 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:33:01.107049 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.107063 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.108117 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.108122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.109697 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.109708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.110942 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.110948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.111123 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.111146 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:33:01.111257 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 10:33:01.111413 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":93,"thread_start_us":281,"threads_started":1}
I20260925 10:33:01.112905 8157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa7156f0a3049ad91b1925aba8ff8ed" candidate_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c557346c1141d58968601009179324" is_pre_election: true
I20260925 10:33:01.113081 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa7156f0a3049ad91b1925aba8ff8ed" candidate_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133464d0ed444863883f61a804eafde6" is_pre_election: true
I20260925 10:33:01.113441 8157 raft_consensus.cc:2454] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a514a91eeaa47428ddd63e5facdd3ae in term 0.
I20260925 10:33:01.113579 8434 raft_consensus.cc:2454] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a514a91eeaa47428ddd63e5facdd3ae in term 0.
I20260925 10:33:01.114064 8224 leader_election.cc:302] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a514a91eeaa47428ddd63e5facdd3ae, e9c557346c1141d58968601009179324; no voters:
I20260925 10:33:01.114488 8510 raft_consensus.cc:2781] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:33:01.114645 8510 raft_consensus.cc:482] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:33:01.114763 8510 raft_consensus.cc:3037] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:01.116981 8510 raft_consensus.cc:504] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } }
I20260925 10:33:01.117527 8510 leader_election.cc:288] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [CANDIDATE]: Term 1 election: Requested vote from peers 133464d0ed444863883f61a804eafde6 (127.0.97.3:40741), e9c557346c1141d58968601009179324 (127.0.97.1:39373)
I20260925 10:33:01.118024 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa7156f0a3049ad91b1925aba8ff8ed" candidate_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133464d0ed444863883f61a804eafde6"
I20260925 10:33:01.118124 8157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa7156f0a3049ad91b1925aba8ff8ed" candidate_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c557346c1141d58968601009179324"
I20260925 10:33:01.118266 8434 raft_consensus.cc:3037] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:01.118350 8157 raft_consensus.cc:3037] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:01.120453 8157 raft_consensus.cc:2454] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a514a91eeaa47428ddd63e5facdd3ae in term 1.
I20260925 10:33:01.120457 8434 raft_consensus.cc:2454] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a514a91eeaa47428ddd63e5facdd3ae in term 1.
I20260925 10:33:01.120818 8224 leader_election.cc:302] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133464d0ed444863883f61a804eafde6, 5a514a91eeaa47428ddd63e5facdd3ae; no voters:
I20260925 10:33:01.121109 8510 raft_consensus.cc:2781] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:33:01.121536 8510 raft_consensus.cc:686] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 LEADER]: Becoming Leader. State: Replica: 5a514a91eeaa47428ddd63e5facdd3ae, State: Running, Role: LEADER
I20260925 10:33:01.122047 8510 consensus_queue.cc:237] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } }
I20260925 10:33:01.126482 7995 catalog_manager.cc:5495] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a514a91eeaa47428ddd63e5facdd3ae (127.0.97.2). New cstate: current_term: 1 leader_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } health_report { overall_health: UNKNOWN } } }
I20260925 10:33:01.149870 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.145130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.0.97.2:44725)
0925 10:33:01.145294 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.145298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.145453 (+ 155us) server_negotiation.cc:407] Connection header received
0925 10:33:01.145618 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.145622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.145692 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.145810 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:33:01.146103 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.146836 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.148037 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.148419 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.148505 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.148953 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:33:01.149443 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:33:01.149559 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.149621 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.149678 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:33:01.157406 8434 raft_consensus.cc:1261] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Refusing update from remote peer 5a514a91eeaa47428ddd63e5facdd3ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:33:01.157567 8157 raft_consensus.cc:1261] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324 [term 1 FOLLOWER]: Refusing update from remote peer 5a514a91eeaa47428ddd63e5facdd3ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 10:33:01.158099 8481 tablet.cc:2369] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:33:01.158408 8515 consensus_queue.cc:1035] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:33:01.158854 8510 consensus_queue.cc:1035] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:33:01.162595 8203 tablet.cc:2369] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:33:01.163834 8342 tablet.cc:2369] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:33:01.168852 8518 mvcc.cc:205] Tried to move back new op lower bound from 7333201433211924480 to 7333201433079373824. Current Snapshot: MvccSnapshot[applied={T|T < 7333201433211924480}]
I20260925 10:33:01.170095 8519 mvcc.cc:205] Tried to move back new op lower bound from 7333201433211924480 to 7333201433079373824. Current Snapshot: MvccSnapshot[applied={T|T < 7333201433211924480}]
I20260925 10:33:01.172401 8520 mvcc.cc:205] Tried to move back new op lower bound from 7333201433211924480 to 7333201433079373824. Current Snapshot: MvccSnapshot[applied={T|T < 7333201433211924480}]
I20260925 10:33:01.182134 7994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:33:01.182543 7994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:33:01.186172 7994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89: Sending DeleteTablet for 3 replicas of tablet dfa7156f0a3049ad91b1925aba8ff8ed
I20260925 10:33:01.187405 8132 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa7156f0a3049ad91b1925aba8ff8ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:33:01 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40166
I20260925 10:33:01.187458 8410 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa7156f0a3049ad91b1925aba8ff8ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:33:01 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48776
I20260925 10:33:01.188141 8271 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa7156f0a3049ad91b1925aba8ff8ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:33:01 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42344
I20260925 10:33:01.189008 7994 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:33:01.189882 7994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:33:01.192590 8529 tablet_replica.cc:330] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae: stopping tablet replica
I20260925 10:33:01.193038 8529 raft_consensus.cc:2229] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:33:01.193715 8529 raft_consensus.cc:2258] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:33:01.194510 8528 tablet_replica.cc:330] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6: stopping tablet replica
I20260925 10:33:01.194931 8528 raft_consensus.cc:2229] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:33:01.195441 8528 raft_consensus.cc:2258] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:33:01.197610 8528 ts_tablet_manager.cc:1905] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:33:01.198400 8529 ts_tablet_manager.cc:1905] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:33:01.199743 8131 tablet_service.cc:1467] Processing CreateTablet for tablet 19b88d38cb374941812c1bae25730569 (DEFAULT_TABLE table=test-table [id=b0d8b6edfe4449268b9131a5a94f5943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:33:01.200255 8131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b88d38cb374941812c1bae25730569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:01.202499 8271 tablet_service.cc:1467] Processing CreateTablet for tablet 19b88d38cb374941812c1bae25730569 (DEFAULT_TABLE table=test-table [id=b0d8b6edfe4449268b9131a5a94f5943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:33:01.202953 8271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b88d38cb374941812c1bae25730569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:01.205377 8528 ts_tablet_manager.cc:1918] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:33:01.205581 8528 log.cc:1196] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/dfa7156f0a3049ad91b1925aba8ff8ed
I20260925 10:33:01.205942 8529 ts_tablet_manager.cc:1918] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:33:01.206099 8529 log.cc:1196] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/dfa7156f0a3049ad91b1925aba8ff8ed
I20260925 10:33:01.206336 8528 ts_tablet_manager.cc:1939] T dfa7156f0a3049ad91b1925aba8ff8ed P 133464d0ed444863883f61a804eafde6: Deleting consensus metadata
I20260925 10:33:01.206683 8529 ts_tablet_manager.cc:1939] T dfa7156f0a3049ad91b1925aba8ff8ed P 5a514a91eeaa47428ddd63e5facdd3ae: Deleting consensus metadata
I20260925 10:33:01.208184 8410 tablet_service.cc:1467] Processing CreateTablet for tablet 19b88d38cb374941812c1bae25730569 (DEFAULT_TABLE table=test-table [id=b0d8b6edfe4449268b9131a5a94f5943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:33:01.208266 7982 catalog_manager.cc:4841] TS 5a514a91eeaa47428ddd63e5facdd3ae (127.0.97.2:44725): tablet dfa7156f0a3049ad91b1925aba8ff8ed (table test-table [id=9ba55f70c6df4bd89adac43b1e2cebe7]) successfully deleted
I20260925 10:33:01.208652 8410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b88d38cb374941812c1bae25730569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:01.209280 8527 tablet_replica.cc:330] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324: stopping tablet replica
I20260925 10:33:01.209712 8527 raft_consensus.cc:2229] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:33:01.211939 8504 tablet_bootstrap.cc:492] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae: Bootstrap starting.
I20260925 10:33:01.212605 7981 catalog_manager.cc:4841] TS 133464d0ed444863883f61a804eafde6 (127.0.97.3:40741): tablet dfa7156f0a3049ad91b1925aba8ff8ed (table test-table [id=9ba55f70c6df4bd89adac43b1e2cebe7]) successfully deleted
I20260925 10:33:01.213656 8504 tablet_bootstrap.cc:654] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:01.214031 8527 raft_consensus.cc:2258] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:33:01.215003 8506 tablet_bootstrap.cc:492] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6: Bootstrap starting.
I20260925 10:33:01.216506 8506 tablet_bootstrap.cc:654] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:01.221300 8527 ts_tablet_manager.cc:1905] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:33:01.224859 8505 tablet_bootstrap.cc:492] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324: Bootstrap starting.
I20260925 10:33:01.225274 8504 tablet_bootstrap.cc:492] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae: No bootstrap required, opened a new log
I20260925 10:33:01.225459 8504 ts_tablet_manager.cc:1397] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260925 10:33:01.226519 8505 tablet_bootstrap.cc:654] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:01.226409 8504 raft_consensus.cc:348] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.226738 8504 raft_consensus.cc:374] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:01.226867 8504 raft_consensus.cc:729] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a514a91eeaa47428ddd63e5facdd3ae, State: Initialized, Role: FOLLOWER
I20260925 10:33:01.227200 8504 consensus_queue.cc:260] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.230382 8504 ts_tablet_manager.cc:1428] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260925 10:33:01.231446 8527 ts_tablet_manager.cc:1918] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:33:01.231676 8527 log.cc:1196] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/dfa7156f0a3049ad91b1925aba8ff8ed
I20260925 10:33:01.232287 8527 ts_tablet_manager.cc:1939] T dfa7156f0a3049ad91b1925aba8ff8ed P e9c557346c1141d58968601009179324: Deleting consensus metadata
I20260925 10:33:01.233754 8505 tablet_bootstrap.cc:492] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324: No bootstrap required, opened a new log
I20260925 10:33:01.233903 8505 ts_tablet_manager.cc:1397] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260925 10:33:01.234274 7981 catalog_manager.cc:4841] TS e9c557346c1141d58968601009179324 (127.0.97.1:39373): tablet dfa7156f0a3049ad91b1925aba8ff8ed (table test-table [id=9ba55f70c6df4bd89adac43b1e2cebe7]) successfully deleted
I20260925 10:33:01.235853 8505 raft_consensus.cc:348] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.236161 8505 raft_consensus.cc:374] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:01.236272 8505 raft_consensus.cc:729] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c557346c1141d58968601009179324, State: Initialized, Role: FOLLOWER
I20260925 10:33:01.236562 8505 consensus_queue.cc:260] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.237258 8505 ts_tablet_manager.cc:1428] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 10:33:01.237874 8506 tablet_bootstrap.cc:492] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6: No bootstrap required, opened a new log
I20260925 10:33:01.238031 8506 ts_tablet_manager.cc:1397] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.003s
I20260925 10:33:01.238895 8506 raft_consensus.cc:348] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.239087 8506 raft_consensus.cc:374] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:01.239183 8506 raft_consensus.cc:729] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133464d0ed444863883f61a804eafde6, State: Initialized, Role: FOLLOWER
I20260925 10:33:01.239463 8506 consensus_queue.cc:260] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.240079 8506 ts_tablet_manager.cc:1428] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:33:01.247573 8512 raft_consensus.cc:482] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:33:01.247774 8512 raft_consensus.cc:504] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.249434 8512 leader_election.cc:288] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9c557346c1141d58968601009179324 (127.0.97.1:39373), 5a514a91eeaa47428ddd63e5facdd3ae (127.0.97.2:44725)
I20260925 10:33:01.257010 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.249567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39373 (local address 127.0.97.3:42131)
0925 10:33:01.250298 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:01.250497 (+ 199us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:01.250632 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:01.251105 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:01.251111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:01.251139 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:33:01.251374 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.251385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.252798 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.252807 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.255001 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.255016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.256311 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.256317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.256497 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.256520 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:33:01.256617 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 10:33:01.256739 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":28}
I20260925 10:33:01.257031 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.249815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42131 (local address 127.0.97.1:39373)
0925 10:33:01.250006 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.250012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.250572 (+ 560us) server_negotiation.cc:407] Connection header received
0925 10:33:01.250762 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.250766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.250851 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.250937 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:33:01.251542 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.252653 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.255171 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.256173 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.256303 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.256395 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.256641 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.256758 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:33:01.257958 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b88d38cb374941812c1bae25730569" candidate_uuid: "133464d0ed444863883f61a804eafde6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c557346c1141d58968601009179324" is_pre_election: true
I20260925 10:33:01.258227 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.250073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59019 (local address 127.0.97.2:44725)
0925 10:33:01.250437 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.250441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.252204 (+ 1763us) server_negotiation.cc:407] Connection header received
0925 10:33:01.252334 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.252340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.252421 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.252519 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:33:01.253125 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.253855 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.255786 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.257053 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.257209 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.257295 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.257808 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.257931 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260925 10:33:01.258292 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.249713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44725 (local address 127.0.97.3:59019)
0925 10:33:01.252097 (+ 2384us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:01.252113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:01.252228 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:01.252540 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:01.252544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:01.252566 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:33:01.252969 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.252978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.253972 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.253977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.255629 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.255642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.257457 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.257463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.257640 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.257674 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 10:33:01.257769 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 10:33:01.257893 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2278,"mutex_wait_us":1,"thread_start_us":2101,"threads_started":1}
I20260925 10:33:01.258261 8156 raft_consensus.cc:2454] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133464d0ed444863883f61a804eafde6 in term 0.
I20260925 10:33:01.259060 8363 leader_election.cc:302] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133464d0ed444863883f61a804eafde6, e9c557346c1141d58968601009179324; no voters:
I20260925 10:33:01.259089 8291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b88d38cb374941812c1bae25730569" candidate_uuid: "133464d0ed444863883f61a804eafde6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" is_pre_election: true
I20260925 10:33:01.259406 8512 raft_consensus.cc:2781] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:33:01.259505 8512 raft_consensus.cc:482] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:33:01.259563 8512 raft_consensus.cc:3037] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:01.259531 8291 raft_consensus.cc:2454] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133464d0ed444863883f61a804eafde6 in term 0.
I20260925 10:33:01.262470 8512 raft_consensus.cc:504] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.263128 8512 leader_election.cc:288] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [CANDIDATE]: Term 1 election: Requested vote from peers e9c557346c1141d58968601009179324 (127.0.97.1:39373), 5a514a91eeaa47428ddd63e5facdd3ae (127.0.97.2:44725)
I20260925 10:33:01.263470 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b88d38cb374941812c1bae25730569" candidate_uuid: "133464d0ed444863883f61a804eafde6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c557346c1141d58968601009179324"
I20260925 10:33:01.263666 8156 raft_consensus.cc:3037] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:01.263648 8291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b88d38cb374941812c1bae25730569" candidate_uuid: "133464d0ed444863883f61a804eafde6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a514a91eeaa47428ddd63e5facdd3ae"
I20260925 10:33:01.263841 8291 raft_consensus.cc:3037] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:01.266700 8156 raft_consensus.cc:2454] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133464d0ed444863883f61a804eafde6 in term 1.
I20260925 10:33:01.267052 8363 leader_election.cc:302] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133464d0ed444863883f61a804eafde6, e9c557346c1141d58968601009179324; no voters:
I20260925 10:33:01.267302 8291 raft_consensus.cc:2454] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133464d0ed444863883f61a804eafde6 in term 1.
I20260925 10:33:01.267355 8512 raft_consensus.cc:2781] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:33:01.270566 8512 raft_consensus.cc:686] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [term 1 LEADER]: Becoming Leader. State: Replica: 133464d0ed444863883f61a804eafde6, State: Running, Role: LEADER
I20260925 10:33:01.270970 8512 consensus_queue.cc:237] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.274458 7994 catalog_manager.cc:5495] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 133464d0ed444863883f61a804eafde6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "133464d0ed444863883f61a804eafde6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } health_report { overall_health: HEALTHY } } }
Sep 25 10:33:01 dist-test-slave-dxdd krb5kdc[7950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332379, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260925 10:33:01.335913 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.320862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39929 (local address 127.0.0.1:49002)
0925 10:33:01.321584 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:01.321600 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:01.321722 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:01.322059 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:01.322062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:01.322801 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:01.323066 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.323075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.324528 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.324532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.325591 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.325602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.326607 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.326612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.326741 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.327412 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:01.327428 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:01.329686 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:01.333447 (+ 3761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:01.333458 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:01.333470 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:01.333868 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:01.334241 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:01.334244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:01.334247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:01.334427 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:01.334995 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:01.335001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:01.335156 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 10:33:01.335286 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 10:33:01.335490 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":3,"thread_start_us":392,"threads_started":1}
I20260925 10:33:01.336107 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.320862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49002 (local address 127.0.97.62:39929)
0925 10:33:01.321112 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.321119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.321688 (+ 569us) server_negotiation.cc:407] Connection header received
0925 10:33:01.321836 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.321843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.321921 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.322038 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:01.323226 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.324379 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.325784 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.326437 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.326571 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.329883 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:01.329890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:01.329895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:01.329932 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:01.333317 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:01.333969 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:01.333975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:01.333979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:01.334132 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:01.334524 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:01.334529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:01.334532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:01.334876 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:01.335019 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.335410 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.335634 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":113}
I20260925 10:33:01.340986 7994 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49002:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:33:01.341976 7994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:33:01.348526 8131 tablet_service.cc:1467] Processing CreateTablet for tablet d7b13d0a87c64bada95c0b241c73a28c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d16d9c240e7454f90700852497a4e3e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:33:01.349072 8131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7b13d0a87c64bada95c0b241c73a28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:01.351141 8271 tablet_service.cc:1467] Processing CreateTablet for tablet d7b13d0a87c64bada95c0b241c73a28c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d16d9c240e7454f90700852497a4e3e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:33:01.351723 8271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7b13d0a87c64bada95c0b241c73a28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:01.354763 8410 tablet_service.cc:1467] Processing CreateTablet for tablet d7b13d0a87c64bada95c0b241c73a28c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d16d9c240e7454f90700852497a4e3e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:33:01.355082 8505 tablet_bootstrap.cc:492] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324: Bootstrap starting.
I20260925 10:33:01.355262 8410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7b13d0a87c64bada95c0b241c73a28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:01.356381 8505 tablet_bootstrap.cc:654] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:01.359616 8504 tablet_bootstrap.cc:492] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae: Bootstrap starting.
I20260925 10:33:01.359543 8505 tablet_bootstrap.cc:492] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324: No bootstrap required, opened a new log
I20260925 10:33:01.359922 8505 ts_tablet_manager.cc:1397] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260925 10:33:01.360663 8505 raft_consensus.cc:348] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.360971 8505 raft_consensus.cc:374] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:01.361047 8505 raft_consensus.cc:729] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c557346c1141d58968601009179324, State: Initialized, Role: FOLLOWER
I20260925 10:33:01.361359 8504 tablet_bootstrap.cc:654] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:01.361394 8505 consensus_queue.cc:260] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.361958 8511 tablet_replica.cc:439] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } }
I20260925 10:33:01.362323 8506 tablet_bootstrap.cc:492] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6: Bootstrap starting.
I20260925 10:33:01.362339 8511 tablet_replica.cc:442] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:33:01.363466 8505 ts_tablet_manager.cc:1428] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 10:33:01.363879 8506 tablet_bootstrap.cc:654] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:01.364979 8504 tablet_bootstrap.cc:492] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae: No bootstrap required, opened a new log
I20260925 10:33:01.365145 8504 ts_tablet_manager.cc:1397] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260925 10:33:01.365724 8504 raft_consensus.cc:348] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.365942 8504 raft_consensus.cc:374] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:01.366036 8504 raft_consensus.cc:729] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a514a91eeaa47428ddd63e5facdd3ae, State: Initialized, Role: FOLLOWER
I20260925 10:33:01.366324 8504 consensus_queue.cc:260] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.366748 8510 tablet_replica.cc:439] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } }
I20260925 10:33:01.366894 8504 ts_tablet_manager.cc:1428] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:33:01.367038 8510 tablet_replica.cc:442] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:33:01.369148 8506 tablet_bootstrap.cc:492] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6: No bootstrap required, opened a new log
I20260925 10:33:01.369324 8506 ts_tablet_manager.cc:1397] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260925 10:33:01.369875 8506 raft_consensus.cc:348] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.370101 8506 raft_consensus.cc:374] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:01.370193 8506 raft_consensus.cc:729] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133464d0ed444863883f61a804eafde6, State: Initialized, Role: FOLLOWER
I20260925 10:33:01.370442 8506 consensus_queue.cc:260] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.370810 8512 tablet_replica.cc:439] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } }
I20260925 10:33:01.370980 8506 ts_tablet_manager.cc:1428] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:33:01.371091 8512 tablet_replica.cc:442] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:33:01.422609 8511 raft_consensus.cc:482] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:33:01.422854 8511 raft_consensus.cc:504] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.424579 8511 leader_election.cc:288] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a514a91eeaa47428ddd63e5facdd3ae (127.0.97.2:44725), 133464d0ed444863883f61a804eafde6 (127.0.97.3:40741)
I20260925 10:33:01.443583 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.425884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38569 (local address 127.0.97.3:40741)
0925 10:33:01.426049 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.426056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.436481 (+ 10425us) server_negotiation.cc:407] Connection header received
0925 10:33:01.436708 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.436714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.436781 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.436834 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:33:01.437570 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.438689 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.441267 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.442426 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.442518 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.442608 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.443173 (+ 565us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.443308 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:33:01.443917 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.425021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46419 (local address 127.0.97.2:44725)
0925 10:33:01.425241 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.425247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.435676 (+ 10429us) server_negotiation.cc:407] Connection header received
0925 10:33:01.435854 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.435861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.435944 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.436040 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:33:01.437150 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.438474 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.441265 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.442688 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.442787 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.442920 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.443518 (+ 598us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.443668 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:33:01.443867 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.425531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40741 (local address 127.0.97.1:38569)
0925 10:33:01.436385 (+ 10854us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:01.436406 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:01.436568 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:01.437012 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:01.437017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:01.437038 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:33:01.437392 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.437401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.438861 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.438867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.441100 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.441116 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.442792 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.442798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.442993 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.443030 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 10:33:01.443133 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 10:33:01.443257 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10746,"mutex_wait_us":31,"thread_start_us":10663,"threads_started":1}
I20260925 10:33:01.444576 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.424777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44725 (local address 127.0.97.1:46419)
0925 10:33:01.435580 (+ 10803us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:01.435599 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:01.435734 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:01.436451 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:01.436456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:01.436549 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:33:01.436927 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.436944 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.438683 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.438688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.441081 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.441095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.443122 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.443130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.443275 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.443301 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 10:33:01.444078 (+ 777us) client_negotiation.cc:240] Negotiation successful
0925 10:33:01.444199 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10674,"mutex_wait_us":66,"thread_start_us":11081,"threads_started":1}
I20260925 10:33:01.444839 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b13d0a87c64bada95c0b241c73a28c" candidate_uuid: "e9c557346c1141d58968601009179324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133464d0ed444863883f61a804eafde6" is_pre_election: true
I20260925 10:33:01.445092 8434 raft_consensus.cc:2454] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c557346c1141d58968601009179324 in term 0.
I20260925 10:33:01.445303 8291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b13d0a87c64bada95c0b241c73a28c" candidate_uuid: "e9c557346c1141d58968601009179324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" is_pre_election: true
I20260925 10:33:01.445580 8291 raft_consensus.cc:2454] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c557346c1141d58968601009179324 in term 0.
I20260925 10:33:01.445545 8084 leader_election.cc:302] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133464d0ed444863883f61a804eafde6, e9c557346c1141d58968601009179324; no voters:
I20260925 10:33:01.445928 8511 raft_consensus.cc:2781] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:33:01.446048 8511 raft_consensus.cc:482] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:33:01.446139 8511 raft_consensus.cc:3037] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:01.449304 8511 raft_consensus.cc:504] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.449965 8511 leader_election.cc:288] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [CANDIDATE]: Term 1 election: Requested vote from peers 5a514a91eeaa47428ddd63e5facdd3ae (127.0.97.2:44725), 133464d0ed444863883f61a804eafde6 (127.0.97.3:40741)
I20260925 10:33:01.450358 8291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b13d0a87c64bada95c0b241c73a28c" candidate_uuid: "e9c557346c1141d58968601009179324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a514a91eeaa47428ddd63e5facdd3ae"
I20260925 10:33:01.450595 8291 raft_consensus.cc:3037] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:01.450897 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b13d0a87c64bada95c0b241c73a28c" candidate_uuid: "e9c557346c1141d58968601009179324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133464d0ed444863883f61a804eafde6"
I20260925 10:33:01.451097 8434 raft_consensus.cc:3037] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:01.453001 8291 raft_consensus.cc:2454] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c557346c1141d58968601009179324 in term 1.
I20260925 10:33:01.453392 8085 leader_election.cc:302] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a514a91eeaa47428ddd63e5facdd3ae, e9c557346c1141d58968601009179324; no voters:
I20260925 10:33:01.453684 8511 raft_consensus.cc:2781] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:33:01.453999 8434 raft_consensus.cc:2454] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c557346c1141d58968601009179324 in term 1.
I20260925 10:33:01.454270 8511 raft_consensus.cc:686] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [term 1 LEADER]: Becoming Leader. State: Replica: e9c557346c1141d58968601009179324, State: Running, Role: LEADER
I20260925 10:33:01.454681 8511 consensus_queue.cc:237] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.456543 8547 tablet_replica.cc:439] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324: TxnStatusTablet state changed. Reason: New leader e9c557346c1141d58968601009179324. Latest consensus state: current_term: 1 leader_uuid: "e9c557346c1141d58968601009179324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } }
I20260925 10:33:01.457017 8547 tablet_replica.cc:442] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:33:01.457796 8549 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:33:01.458016 8549 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:33:01.458252 7993 catalog_manager.cc:5495] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9c557346c1141d58968601009179324 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e9c557346c1141d58968601009179324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } health_report { overall_health: UNKNOWN } } }
I20260925 10:33:01.486986 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.479738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38922 (local address 127.0.97.1:39373)
0925 10:33:01.479916 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.479922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.480122 (+ 200us) server_negotiation.cc:407] Connection header received
0925 10:33:01.480386 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.480393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.480510 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.480655 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:33:01.481067 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.482237 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.484174 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.484849 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.485000 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.485523 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:33:01.486283 (+ 760us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:33:01.486465 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.486557 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.486645 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:33:01.486997 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.479836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39373 (local address 127.0.0.1:38922)
0925 10:33:01.480027 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:01.480046 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:01.480188 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:01.480688 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:01.480693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:01.480706 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:33:01.480889 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.480900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.482398 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.482403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.484026 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.484040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.485027 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.485034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.485243 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.485311 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:33:01.486416 (+ 1105us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:33:01.486444 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 10:33:01.486512 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 10:33:01.486666 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 10:33:01.491694 8434 raft_consensus.cc:1261] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Refusing update from remote peer e9c557346c1141d58968601009179324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:33:01.492105 8291 raft_consensus.cc:1261] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Refusing update from remote peer e9c557346c1141d58968601009179324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:33:01.492722 8511 consensus_queue.cc:1035] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:33:01.493360 8511 consensus_queue.cc:1035] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:33:01.504056 8510 tablet_replica.cc:439] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae: TxnStatusTablet state changed. Reason: New leader e9c557346c1141d58968601009179324. Latest consensus state: current_term: 1 leader_uuid: "e9c557346c1141d58968601009179324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } }
I20260925 10:33:01.504335 8510 tablet_replica.cc:442] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:33:01.505214 8512 tablet_replica.cc:439] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6: TxnStatusTablet state changed. Reason: New leader e9c557346c1141d58968601009179324. Latest consensus state: current_term: 1 leader_uuid: "e9c557346c1141d58968601009179324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } }
I20260925 10:33:01.505491 8512 tablet_replica.cc:442] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:33:01.510499 8510 tablet_replica.cc:439] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9c557346c1141d58968601009179324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } }
I20260925 10:33:01.510757 8510 tablet_replica.cc:442] T d7b13d0a87c64bada95c0b241c73a28c P 5a514a91eeaa47428ddd63e5facdd3ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:33:01.517735 8555 tablet_replica.cc:439] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9c557346c1141d58968601009179324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } }
I20260925 10:33:01.518182 8555 tablet_replica.cc:442] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:33:01.524442 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.517122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45926 (local address 127.0.97.3:40741)
0925 10:33:01.517275 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.517281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.517298 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:33:01.517390 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.517394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.517497 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.517650 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:33:01.518284 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.519184 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.521075 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.521770 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.521891 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.522584 (+ 693us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:33:01.523293 (+ 709us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:33:01.523427 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.524008 (+ 581us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.524090 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:33:01.525077 8511 tablet_replica.cc:439] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9c557346c1141d58968601009179324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } }
I20260925 10:33:01.525385 8511 tablet_replica.cc:442] T d7b13d0a87c64bada95c0b241c73a28c P e9c557346c1141d58968601009179324: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:33:01.529412 8512 tablet_replica.cc:439] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9c557346c1141d58968601009179324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } } }
I20260925 10:33:01.529692 8512 tablet_replica.cc:442] T d7b13d0a87c64bada95c0b241c73a28c P 133464d0ed444863883f61a804eafde6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:33:01.553580 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.540730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57909 (local address 127.0.97.1:39373)
0925 10:33:01.540937 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.540944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.546979 (+ 6035us) server_negotiation.cc:407] Connection header received
0925 10:33:01.547311 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.547317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.547481 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.547599 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:33:01.548166 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.549339 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.551402 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.552115 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.552234 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.552758 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:33:01.552925 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:33:01.553013 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.553171 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.553249 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 10:33:01.553934 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.540613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39373 (local address 127.0.97.3:57909)
0925 10:33:01.546847 (+ 6234us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:01.546868 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:01.547124 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:01.547665 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:01.547670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:01.547684 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:33:01.548002 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.548014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.549496 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.549502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.551252 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.551267 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.552301 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.552308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.552496 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.552571 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:33:01.553033 (+ 462us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:33:01.553047 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 10:33:01.553139 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 10:33:01.553288 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6036,"thread_start_us":5904,"threads_started":1}
I20260925 10:33:01.565130 8291 raft_consensus.cc:1261] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Refusing update from remote peer 133464d0ed444863883f61a804eafde6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:33:01.565289 8156 raft_consensus.cc:1261] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324 [term 1 FOLLOWER]: Refusing update from remote peer 133464d0ed444863883f61a804eafde6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:33:01.565933 8534 consensus_queue.cc:1035] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:33:01.566425 8512 consensus_queue.cc:1035] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:33:01.627125 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.613763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42089 (local address 127.0.97.3:40741)
0925 10:33:01.613940 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.613946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.620555 (+ 6609us) server_negotiation.cc:407] Connection header received
0925 10:33:01.620722 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.620727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.620819 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.620922 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:33:01.621556 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.622732 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.624903 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.625605 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.625766 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.626298 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:33:01.626479 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:33:01.626567 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.626720 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.626810 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:33:01.627537 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.613690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40741 (local address 127.0.97.1:42089)
0925 10:33:01.620408 (+ 6718us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:01.620429 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:01.620580 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:01.620960 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:01.620965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:01.620980 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:33:01.621385 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.621396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.622901 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.622907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.624765 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:01.624781 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.625828 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.625835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:01.626018 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.626108 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:33:01.626585 (+ 477us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:33:01.626599 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 10:33:01.626690 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 10:33:01.626851 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6497,"thread_start_us":6339,"threads_started":1}
I20260925 10:33:01.688117 7993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:33:01.688364 7993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:33:01.691794 7993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89: Sending DeleteTablet for 3 replicas of tablet 19b88d38cb374941812c1bae25730569
I20260925 10:33:01.692656 8271 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b88d38cb374941812c1bae25730569 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:33:01 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42344
I20260925 10:33:01.692920 8529 tablet_replica.cc:330] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae: stopping tablet replica
I20260925 10:33:01.693166 8529 raft_consensus.cc:2229] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:33:01.693339 8529 raft_consensus.cc:2258] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:33:01.694054 8410 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b88d38cb374941812c1bae25730569 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:33:01 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48776
I20260925 10:33:01.694329 8528 tablet_replica.cc:330] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6: stopping tablet replica
I20260925 10:33:01.694540 8528 raft_consensus.cc:2229] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:33:01.694981 8528 raft_consensus.cc:2258] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:33:01.695585 8131 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b88d38cb374941812c1bae25730569 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:33:01 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40166
I20260925 10:33:01.695856 8527 tablet_replica.cc:330] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324: stopping tablet replica
I20260925 10:33:01.696067 8527 raft_consensus.cc:2229] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:33:01.696240 8527 raft_consensus.cc:2258] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:33:01.696811 8529 ts_tablet_manager.cc:1905] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:33:01.699548 8527 ts_tablet_manager.cc:1905] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:33:01.700130 8528 ts_tablet_manager.cc:1905] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:33:01.707571 8528 ts_tablet_manager.cc:1918] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:33:01.712481 8527 ts_tablet_manager.cc:1918] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:33:01.713617 8529 ts_tablet_manager.cc:1918] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:33:01.713800 8529 log.cc:1196] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/19b88d38cb374941812c1bae25730569
I20260925 10:33:01.714237 8529 ts_tablet_manager.cc:1939] T 19b88d38cb374941812c1bae25730569 P 5a514a91eeaa47428ddd63e5facdd3ae: Deleting consensus metadata
I20260925 10:33:01.714419 8528 log.cc:1196] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/19b88d38cb374941812c1bae25730569
I20260925 10:33:01.719342 8527 log.cc:1196] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/19b88d38cb374941812c1bae25730569
I20260925 10:33:01.719929 8527 ts_tablet_manager.cc:1939] T 19b88d38cb374941812c1bae25730569 P e9c557346c1141d58968601009179324: Deleting consensus metadata
I20260925 10:33:01.720435 8528 ts_tablet_manager.cc:1939] T 19b88d38cb374941812c1bae25730569 P 133464d0ed444863883f61a804eafde6: Deleting consensus metadata
I20260925 10:33:01.721239 7981 catalog_manager.cc:4841] TS e9c557346c1141d58968601009179324 (127.0.97.1:39373): tablet 19b88d38cb374941812c1bae25730569 (table test-table [id=b0d8b6edfe4449268b9131a5a94f5943]) successfully deleted
I20260925 10:33:01.721820 7981 catalog_manager.cc:4841] TS 133464d0ed444863883f61a804eafde6 (127.0.97.3:40741): tablet 19b88d38cb374941812c1bae25730569 (table test-table [id=b0d8b6edfe4449268b9131a5a94f5943]) successfully deleted
I20260925 10:33:01.722235 7982 catalog_manager.cc:4841] TS 5a514a91eeaa47428ddd63e5facdd3ae (127.0.97.2:44725): tablet 19b88d38cb374941812c1bae25730569 (table test-table [id=b0d8b6edfe4449268b9131a5a94f5943]) successfully deleted
Sep 25 10:33:01 dist-test-slave-dxdd krb5kdc[7950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:33:01 dist-test-slave-dxdd krb5kdc[7950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:01 dist-test-slave-dxdd krb5kdc[7950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:33:01.800324 8090 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:01.776013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.0.97.62:39929)
0925 10:33:01.776188 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.776195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.776767 (+ 572us) server_negotiation.cc:407] Connection header received
0925 10:33:01.776975 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.776981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.777069 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.777193 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:01.779258 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.780499 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.782592 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.783226 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.783354 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.800034 (+ 16680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39942: BlockingRecv error: recv got EOF from 127.0.0.1:39942 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 10:33:01 dist-test-slave-dxdd krb5kdc[7950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260925 10:33:01.822629 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.804021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39954 (local address 127.0.97.62:39929)
0925 10:33:01.804196 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.804202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.804655 (+ 453us) server_negotiation.cc:407] Connection header received
0925 10:33:01.804912 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.804918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.804998 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.805071 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:01.806965 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.808225 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.810042 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.810799 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.810948 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.815950 (+ 5002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:01.815957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:01.815962 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:01.816005 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:01.819535 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:01.820339 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:01.820346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:01.820350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:01.820518 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:01.820985 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:01.820990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:01.820993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:01.821364 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:01.821630 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.821992 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.822223 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:33:01.826761 7993 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:33:01.827463 7993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:33:01.835557 8271 tablet_service.cc:1467] Processing CreateTablet for tablet 822c6aadf4bc4d18aeb0c66322a0337d (DEFAULT_TABLE table=test-table [id=c88830e663b5406eb7179274ee516118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:33:01.835661 8410 tablet_service.cc:1467] Processing CreateTablet for tablet 822c6aadf4bc4d18aeb0c66322a0337d (DEFAULT_TABLE table=test-table [id=c88830e663b5406eb7179274ee516118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:33:01.836014 8271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822c6aadf4bc4d18aeb0c66322a0337d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:01.836097 8410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822c6aadf4bc4d18aeb0c66322a0337d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:01.843276 8131 tablet_service.cc:1467] Processing CreateTablet for tablet 822c6aadf4bc4d18aeb0c66322a0337d (DEFAULT_TABLE table=test-table [id=c88830e663b5406eb7179274ee516118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:33:01.843745 8131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822c6aadf4bc4d18aeb0c66322a0337d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:01.846549 8504 tablet_bootstrap.cc:492] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae: Bootstrap starting.
I20260925 10:33:01.848228 8504 tablet_bootstrap.cc:654] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:01.853700 8505 tablet_bootstrap.cc:492] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324: Bootstrap starting.
I20260925 10:33:01.855389 8505 tablet_bootstrap.cc:654] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:01.856563 8506 tablet_bootstrap.cc:492] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6: Bootstrap starting.
I20260925 10:33:01.858233 8506 tablet_bootstrap.cc:654] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:01.861938 8504 tablet_bootstrap.cc:492] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae: No bootstrap required, opened a new log
I20260925 10:33:01.862094 8504 ts_tablet_manager.cc:1397] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.005s
I20260925 10:33:01.862895 8504 raft_consensus.cc:348] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.863196 8504 raft_consensus.cc:374] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:01.863299 8504 raft_consensus.cc:729] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a514a91eeaa47428ddd63e5facdd3ae, State: Initialized, Role: FOLLOWER
I20260925 10:33:01.863562 8504 consensus_queue.cc:260] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.864135 8504 ts_tablet_manager.cc:1428] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:33:01.870767 8506 tablet_bootstrap.cc:492] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6: No bootstrap required, opened a new log
I20260925 10:33:01.870991 8506 ts_tablet_manager.cc:1397] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260925 10:33:01.871881 8506 raft_consensus.cc:348] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.872181 8505 tablet_bootstrap.cc:492] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324: No bootstrap required, opened a new log
I20260925 10:33:01.872217 8506 raft_consensus.cc:374] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:01.872329 8505 ts_tablet_manager.cc:1397] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.009s
I20260925 10:33:01.872449 8506 raft_consensus.cc:729] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133464d0ed444863883f61a804eafde6, State: Initialized, Role: FOLLOWER
I20260925 10:33:01.872769 8506 consensus_queue.cc:260] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.873135 8505 raft_consensus.cc:348] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.873420 8505 raft_consensus.cc:374] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:01.874030 8505 raft_consensus.cc:729] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c557346c1141d58968601009179324, State: Initialized, Role: FOLLOWER
I20260925 10:33:01.874294 8505 consensus_queue.cc:260] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.874918 8505 ts_tablet_manager.cc:1428] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 10:33:01.876369 8506 ts_tablet_manager.cc:1428] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260925 10:33:01.904394 8555 raft_consensus.cc:482] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:33:01.904626 8555 raft_consensus.cc:504] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.905447 8555 leader_election.cc:288] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a514a91eeaa47428ddd63e5facdd3ae (127.0.97.2:44725), 133464d0ed444863883f61a804eafde6 (127.0.97.3:40741)
I20260925 10:33:01.906219 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822c6aadf4bc4d18aeb0c66322a0337d" candidate_uuid: "e9c557346c1141d58968601009179324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133464d0ed444863883f61a804eafde6" is_pre_election: true
I20260925 10:33:01.906508 8434 raft_consensus.cc:2454] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c557346c1141d58968601009179324 in term 0.
I20260925 10:33:01.907401 8291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822c6aadf4bc4d18aeb0c66322a0337d" candidate_uuid: "e9c557346c1141d58968601009179324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" is_pre_election: true
I20260925 10:33:01.907656 8291 raft_consensus.cc:2454] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c557346c1141d58968601009179324 in term 0.
I20260925 10:33:01.910637 8084 leader_election.cc:302] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133464d0ed444863883f61a804eafde6, e9c557346c1141d58968601009179324; no voters:
I20260925 10:33:01.911101 8555 raft_consensus.cc:2781] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:33:01.911227 8555 raft_consensus.cc:482] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:33:01.911341 8555 raft_consensus.cc:3037] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:01.914552 8555 raft_consensus.cc:504] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.915612 8291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822c6aadf4bc4d18aeb0c66322a0337d" candidate_uuid: "e9c557346c1141d58968601009179324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a514a91eeaa47428ddd63e5facdd3ae"
I20260925 10:33:01.915819 8291 raft_consensus.cc:3037] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:01.922811 8291 raft_consensus.cc:2454] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c557346c1141d58968601009179324 in term 1.
I20260925 10:33:01.923257 8085 leader_election.cc:302] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a514a91eeaa47428ddd63e5facdd3ae, e9c557346c1141d58968601009179324; no voters:
I20260925 10:33:01.923579 8511 raft_consensus.cc:2781] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:33:01.923758 8511 raft_consensus.cc:686] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [term 1 LEADER]: Becoming Leader. State: Replica: e9c557346c1141d58968601009179324, State: Running, Role: LEADER
I20260925 10:33:01.924011 8511 consensus_queue.cc:237] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:01.927114 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822c6aadf4bc4d18aeb0c66322a0337d" candidate_uuid: "e9c557346c1141d58968601009179324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133464d0ed444863883f61a804eafde6"
I20260925 10:33:01.927326 8434 raft_consensus.cc:3037] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:01.930554 8434 raft_consensus.cc:2454] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c557346c1141d58968601009179324 in term 1.
I20260925 10:33:01.932842 7995 catalog_manager.cc:5495] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9c557346c1141d58968601009179324 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e9c557346c1141d58968601009179324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } health_report { overall_health: UNKNOWN } } }
I20260925 10:33:01.936757 8555 leader_election.cc:288] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [CANDIDATE]: Term 1 election: Requested vote from peers 5a514a91eeaa47428ddd63e5facdd3ae (127.0.97.2:44725), 133464d0ed444863883f61a804eafde6 (127.0.97.3:40741)
I20260925 10:33:01.954164 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:01.947084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38926 (local address 127.0.97.1:39373)
0925 10:33:01.947257 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:01.947263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:01.947859 (+ 596us) server_negotiation.cc:407] Connection header received
0925 10:33:01.947928 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:01.947933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:01.948013 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:01.948107 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:33:01.948634 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.949692 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.951749 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:01.952388 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:01.952542 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:01.953111 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:33:01.953243 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:33:01.953318 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:01.953723 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 10:33:01.953796 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:33:01.958465 8291 raft_consensus.cc:1261] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Refusing update from remote peer e9c557346c1141d58968601009179324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:33:01.959034 8511 consensus_queue.cc:1035] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:33:01.962095 8434 raft_consensus.cc:1261] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Refusing update from remote peer e9c557346c1141d58968601009179324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:33:01.962632 8511 consensus_queue.cc:1035] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:33:01.985445 7994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:33:01.985700 7994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:33:01.989717 7994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89: Sending DeleteTablet for 3 replicas of tablet 822c6aadf4bc4d18aeb0c66322a0337d
I20260925 10:33:01.990409 8131 tablet_service.cc:1514] Processing DeleteTablet for tablet 822c6aadf4bc4d18aeb0c66322a0337d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:33:01 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40166
I20260925 10:33:01.990706 8527 tablet_replica.cc:330] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324: stopping tablet replica
I20260925 10:33:01.990899 8527 raft_consensus.cc:2229] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:33:01.991442 8527 raft_consensus.cc:2258] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:33:01.991745 8271 tablet_service.cc:1514] Processing DeleteTablet for tablet 822c6aadf4bc4d18aeb0c66322a0337d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:33:01 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42344
I20260925 10:33:01.992028 8529 tablet_replica.cc:330] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae: stopping tablet replica
I20260925 10:33:01.992053 8410 tablet_service.cc:1514] Processing DeleteTablet for tablet 822c6aadf4bc4d18aeb0c66322a0337d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:33:01 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48776
I20260925 10:33:01.992234 8529 raft_consensus.cc:2229] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:33:01.992475 8529 raft_consensus.cc:2258] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:33:01.992678 7992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 10:33:01.993153 8528 tablet_replica.cc:330] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6: stopping tablet replica
I20260925 10:33:01.993311 8528 raft_consensus.cc:2229] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 10:33:01.993340 7992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:33:01.993525 8528 raft_consensus.cc:2258] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:33:01.994565 8527 ts_tablet_manager.cc:1905] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:33:01.994761 8529 ts_tablet_manager.cc:1905] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:33:02.000686 8410 tablet_service.cc:1467] Processing CreateTablet for tablet e255444330c6405d81bb3aa586e1b05d (DEFAULT_TABLE table=test-table [id=683b4206674546ed88dd3bf0e56d0d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:33:02.000967 8527 ts_tablet_manager.cc:1918] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:33:02.001109 8410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e255444330c6405d81bb3aa586e1b05d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:02.001104 8527 log.cc:1196] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/822c6aadf4bc4d18aeb0c66322a0337d
I20260925 10:33:02.001142 8529 ts_tablet_manager.cc:1918] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:33:02.001451 8529 log.cc:1196] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/822c6aadf4bc4d18aeb0c66322a0337d
I20260925 10:33:02.001516 8527 ts_tablet_manager.cc:1939] T 822c6aadf4bc4d18aeb0c66322a0337d P e9c557346c1141d58968601009179324: Deleting consensus metadata
I20260925 10:33:02.001921 8529 ts_tablet_manager.cc:1939] T 822c6aadf4bc4d18aeb0c66322a0337d P 5a514a91eeaa47428ddd63e5facdd3ae: Deleting consensus metadata
I20260925 10:33:02.003154 8271 tablet_service.cc:1467] Processing CreateTablet for tablet e255444330c6405d81bb3aa586e1b05d (DEFAULT_TABLE table=test-table [id=683b4206674546ed88dd3bf0e56d0d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:33:02.003575 8271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e255444330c6405d81bb3aa586e1b05d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:02.003544 7982 catalog_manager.cc:4841] TS 5a514a91eeaa47428ddd63e5facdd3ae (127.0.97.2:44725): tablet 822c6aadf4bc4d18aeb0c66322a0337d (table test-table [id=c88830e663b5406eb7179274ee516118]) successfully deleted
I20260925 10:33:02.004994 8131 tablet_service.cc:1467] Processing CreateTablet for tablet e255444330c6405d81bb3aa586e1b05d (DEFAULT_TABLE table=test-table [id=683b4206674546ed88dd3bf0e56d0d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:33:02.005357 8131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e255444330c6405d81bb3aa586e1b05d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:02.009073 8270 tablet_service.cc:1514] Processing DeleteTablet for tablet 822c6aadf4bc4d18aeb0c66322a0337d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:33:01 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42344
I20260925 10:33:02.009317 8528 ts_tablet_manager.cc:1905] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:33:02.009444 7981 catalog_manager.cc:4841] TS e9c557346c1141d58968601009179324 (127.0.97.1:39373): tablet 822c6aadf4bc4d18aeb0c66322a0337d (table test-table [id=c88830e663b5406eb7179274ee516118]) successfully deleted
W20260925 10:33:02.011354 7982 catalog_manager.cc:4808] TS 5a514a91eeaa47428ddd63e5facdd3ae (127.0.97.2:44725): delete failed for tablet 822c6aadf4bc4d18aeb0c66322a0337d because the tablet was not found. No further retry: Not found: Tablet not found: 822c6aadf4bc4d18aeb0c66322a0337d
I20260925 10:33:02.011592 8505 tablet_bootstrap.cc:492] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324: Bootstrap starting.
I20260925 10:33:02.013134 8505 tablet_bootstrap.cc:654] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:02.013173 8504 tablet_bootstrap.cc:492] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae: Bootstrap starting.
I20260925 10:33:02.014758 8504 tablet_bootstrap.cc:654] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:02.019150 8528 ts_tablet_manager.cc:1918] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:33:02.019280 8528 log.cc:1196] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/822c6aadf4bc4d18aeb0c66322a0337d
I20260925 10:33:02.021060 8506 tablet_bootstrap.cc:492] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6: Bootstrap starting.
I20260925 10:33:02.021898 8528 ts_tablet_manager.cc:1939] T 822c6aadf4bc4d18aeb0c66322a0337d P 133464d0ed444863883f61a804eafde6: Deleting consensus metadata
I20260925 10:33:02.022840 8506 tablet_bootstrap.cc:654] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:02.025450 7981 catalog_manager.cc:4841] TS 133464d0ed444863883f61a804eafde6 (127.0.97.3:40741): tablet 822c6aadf4bc4d18aeb0c66322a0337d (table test-table [id=c88830e663b5406eb7179274ee516118]) successfully deleted
I20260925 10:33:02.026595 8505 tablet_bootstrap.cc:492] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324: No bootstrap required, opened a new log
I20260925 10:33:02.026753 8505 ts_tablet_manager.cc:1397] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.001s
I20260925 10:33:02.027444 8505 raft_consensus.cc:348] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:02.027696 8505 raft_consensus.cc:374] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:02.027766 8505 raft_consensus.cc:729] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c557346c1141d58968601009179324, State: Initialized, Role: FOLLOWER
I20260925 10:33:02.028024 8505 consensus_queue.cc:260] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:02.028542 8505 ts_tablet_manager.cc:1428] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:33:02.035331 8504 tablet_bootstrap.cc:492] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae: No bootstrap required, opened a new log
I20260925 10:33:02.035537 8504 ts_tablet_manager.cc:1397] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.007s
I20260925 10:33:02.036437 8504 raft_consensus.cc:348] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:02.036696 8504 raft_consensus.cc:374] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:02.036773 8504 raft_consensus.cc:729] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a514a91eeaa47428ddd63e5facdd3ae, State: Initialized, Role: FOLLOWER
I20260925 10:33:02.037024 8504 consensus_queue.cc:260] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:02.040678 8504 ts_tablet_manager.cc:1428] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260925 10:33:02.041020 8506 tablet_bootstrap.cc:492] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6: No bootstrap required, opened a new log
I20260925 10:33:02.041239 8506 ts_tablet_manager.cc:1397] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.003s
I20260925 10:33:02.042083 8506 raft_consensus.cc:348] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:02.042394 8506 raft_consensus.cc:374] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:02.042531 8506 raft_consensus.cc:729] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133464d0ed444863883f61a804eafde6, State: Initialized, Role: FOLLOWER
I20260925 10:33:02.042847 8506 consensus_queue.cc:260] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:02.043476 8506 ts_tablet_manager.cc:1428] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:33:02.111245 8555 raft_consensus.cc:482] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:33:02.111469 8555 raft_consensus.cc:504] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:02.112306 8555 leader_election.cc:288] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a514a91eeaa47428ddd63e5facdd3ae (127.0.97.2:44725), 133464d0ed444863883f61a804eafde6 (127.0.97.3:40741)
I20260925 10:33:02.112906 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e255444330c6405d81bb3aa586e1b05d" candidate_uuid: "e9c557346c1141d58968601009179324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" is_pre_election: true
I20260925 10:33:02.113050 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e255444330c6405d81bb3aa586e1b05d" candidate_uuid: "e9c557346c1141d58968601009179324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133464d0ed444863883f61a804eafde6" is_pre_election: true
I20260925 10:33:02.113157 8290 raft_consensus.cc:2454] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c557346c1141d58968601009179324 in term 0.
I20260925 10:33:02.113308 8433 raft_consensus.cc:2454] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c557346c1141d58968601009179324 in term 0.
I20260925 10:33:02.113821 8084 leader_election.cc:302] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133464d0ed444863883f61a804eafde6, e9c557346c1141d58968601009179324; no voters:
I20260925 10:33:02.114158 8555 raft_consensus.cc:2781] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:33:02.114288 8555 raft_consensus.cc:482] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:33:02.114406 8555 raft_consensus.cc:3037] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:02.117527 8555 raft_consensus.cc:504] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:02.118175 8555 leader_election.cc:288] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [CANDIDATE]: Term 1 election: Requested vote from peers 5a514a91eeaa47428ddd63e5facdd3ae (127.0.97.2:44725), 133464d0ed444863883f61a804eafde6 (127.0.97.3:40741)
I20260925 10:33:02.118703 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e255444330c6405d81bb3aa586e1b05d" candidate_uuid: "e9c557346c1141d58968601009179324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a514a91eeaa47428ddd63e5facdd3ae"
I20260925 10:33:02.118907 8290 raft_consensus.cc:3037] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:02.118908 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e255444330c6405d81bb3aa586e1b05d" candidate_uuid: "e9c557346c1141d58968601009179324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133464d0ed444863883f61a804eafde6"
I20260925 10:33:02.119107 8433 raft_consensus.cc:3037] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:02.121910 8290 raft_consensus.cc:2454] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c557346c1141d58968601009179324 in term 1.
I20260925 10:33:02.122112 8433 raft_consensus.cc:2454] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c557346c1141d58968601009179324 in term 1.
I20260925 10:33:02.122392 8085 leader_election.cc:302] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a514a91eeaa47428ddd63e5facdd3ae, e9c557346c1141d58968601009179324; no voters:
I20260925 10:33:02.122772 8555 raft_consensus.cc:2781] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:33:02.123268 8555 raft_consensus.cc:686] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [term 1 LEADER]: Becoming Leader. State: Replica: e9c557346c1141d58968601009179324, State: Running, Role: LEADER
I20260925 10:33:02.123580 8555 consensus_queue.cc:237] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } }
I20260925 10:33:02.127453 7995 catalog_manager.cc:5495] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9c557346c1141d58968601009179324 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e9c557346c1141d58968601009179324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c557346c1141d58968601009179324" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 } health_report { overall_health: UNKNOWN } } }
I20260925 10:33:02.181296 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:02.171160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49591 (local address 127.0.97.1:39373)
0925 10:33:02.171346 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:02.171353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:02.174884 (+ 3531us) server_negotiation.cc:407] Connection header received
0925 10:33:02.175064 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:02.175070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:02.175147 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:02.175241 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:33:02.175801 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:02.177051 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:02.179148 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:02.179864 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:02.179986 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:02.180423 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:33:02.180600 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:33:02.180688 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:02.180844 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 10:33:02.180919 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":52}
I20260925 10:33:02.181365 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:02.171071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39373 (local address 127.0.97.1:49591)
0925 10:33:02.174759 (+ 3688us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:02.174788 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:02.174939 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:02.175297 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:02.175303 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:02.175315 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:33:02.175599 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:02.175614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:02.177229 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:02.177234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:02.179013 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:02.179027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:02.180028 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:02.180035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:02.180193 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:02.180239 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:33:02.180709 (+ 470us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:33:02.180721 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 10:33:02.180815 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 10:33:02.180970 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3532,"thread_start_us":3397,"threads_started":1}
I20260925 10:33:02.192247 8433 raft_consensus.cc:1261] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Refusing update from remote peer e9c557346c1141d58968601009179324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:33:02.192257 8290 raft_consensus.cc:1261] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Refusing update from remote peer e9c557346c1141d58968601009179324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:33:02.192970 8511 consensus_queue.cc:1035] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a514a91eeaa47428ddd63e5facdd3ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:33:02.193641 8555 consensus_queue.cc:1035] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "133464d0ed444863883f61a804eafde6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:33:02.277549 7993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:33:02.277812 7993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:33:02.281190 7993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 773b290855614644aebd3d2521477b89: Sending DeleteTablet for 3 replicas of tablet e255444330c6405d81bb3aa586e1b05d
I20260925 10:33:02.282215 8410 tablet_service.cc:1514] Processing DeleteTablet for tablet e255444330c6405d81bb3aa586e1b05d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:33:02 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48776
I20260925 10:33:02.282258 8271 tablet_service.cc:1514] Processing DeleteTablet for tablet e255444330c6405d81bb3aa586e1b05d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:33:02 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42344
I20260925 10:33:02.282532 8528 tablet_replica.cc:330] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6: stopping tablet replica
I20260925 10:33:02.282553 8529 tablet_replica.cc:330] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae: stopping tablet replica
I20260925 10:33:02.282739 8528 raft_consensus.cc:2229] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:33:02.282749 8529 raft_consensus.cc:2229] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:33:02.282919 8529 raft_consensus.cc:2258] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:33:02.282919 8528 raft_consensus.cc:2258] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:33:02.283725 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 8066
I20260925 10:33:02.284324 8529 ts_tablet_manager.cc:1905] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:33:02.282241 8131 tablet_service.cc:1514] Processing DeleteTablet for tablet e255444330c6405d81bb3aa586e1b05d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:33:02 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40166
I20260925 10:33:02.289537 8527 tablet_replica.cc:330] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324: stopping tablet replica
I20260925 10:33:02.289758 8527 raft_consensus.cc:2229] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:33:02.290071 8527 raft_consensus.cc:2258] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:33:02.291314 8527 ts_tablet_manager.cc:1905] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:33:02.298022 8527 ts_tablet_manager.cc:1918] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:33:02.301265 8528 ts_tablet_manager.cc:1905] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:33:02.305568 8529 ts_tablet_manager.cc:1918] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:33:02.305742 8529 log.cc:1196] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/e255444330c6405d81bb3aa586e1b05d
I20260925 10:33:02.306119 8529 ts_tablet_manager.cc:1939] T e255444330c6405d81bb3aa586e1b05d P 5a514a91eeaa47428ddd63e5facdd3ae: Deleting consensus metadata
I20260925 10:33:02.306432 8527 log.cc:1196] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/e255444330c6405d81bb3aa586e1b05d
I20260925 10:33:02.306844 8527 ts_tablet_manager.cc:1939] T e255444330c6405d81bb3aa586e1b05d P e9c557346c1141d58968601009179324: Deleting consensus metadata
I20260925 10:33:02.308391 8528 ts_tablet_manager.cc:1918] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:33:02.308609 8528 log.cc:1196] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/e255444330c6405d81bb3aa586e1b05d
I20260925 10:33:02.308797 7982 catalog_manager.cc:4841] TS 5a514a91eeaa47428ddd63e5facdd3ae (127.0.97.2:44725): tablet e255444330c6405d81bb3aa586e1b05d (table test-table [id=683b4206674546ed88dd3bf0e56d0d92]) successfully deleted
I20260925 10:33:02.309062 7981 catalog_manager.cc:4841] TS e9c557346c1141d58968601009179324 (127.0.97.1:39373): tablet e255444330c6405d81bb3aa586e1b05d (table test-table [id=683b4206674546ed88dd3bf0e56d0d92]) successfully deleted
I20260925 10:33:02.309325 8528 ts_tablet_manager.cc:1939] T e255444330c6405d81bb3aa586e1b05d P 133464d0ed444863883f61a804eafde6: Deleting consensus metadata
Sep 25 10:33:02 dist-test-slave-dxdd krb5kdc[7950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260925 10:33:02.310604 7981 catalog_manager.cc:4841] TS 133464d0ed444863883f61a804eafde6 (127.0.97.3:40741): tablet e255444330c6405d81bb3aa586e1b05d (table test-table [id=683b4206674546ed88dd3bf0e56d0d92]) successfully deleted
I20260925 10:33:02.317741 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:02.285358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38938 (local address 127.0.97.1:39373)
0925 10:33:02.285546 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:02.285552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:02.295805 (+ 10253us) server_negotiation.cc:407] Connection header received
0925 10:33:02.296070 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:02.296075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:02.296174 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:02.296251 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:02.298011 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:02.299197 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:02.301253 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:02.301974 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:02.302073 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:02.311095 (+ 9022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:02.311111 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:02.311127 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:02.311165 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:02.314202 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:02.314883 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:02.314893 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:02.314901 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:02.315052 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:02.315415 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:02.315424 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:02.315426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:02.315807 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:02.316176 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:02.317029 (+ 853us) server_negotiation.cc:299] Negotiation successful
0925 10:33:02.317303 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:33:02.320116 8197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:02.502758 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 8066
I20260925 10:33:02.522276 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 8206
Sep 25 10:33:02 dist-test-slave-dxdd krb5kdc[7950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260925 10:33:02.536324 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:02.523051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.0.97.2:44725)
0925 10:33:02.523529 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:02.523533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:02.523564 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:33:02.523619 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:02.523622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:02.523679 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:02.523769 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:02.524891 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:02.525605 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:02.526824 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:02.527251 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:02.527351 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:02.531289 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:02.531302 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:02.531318 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:02.531350 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:02.533843 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:02.534514 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:02.534521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:02.534523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:02.534657 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:02.535042 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:02.535046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:02.535048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:02.535351 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:02.535566 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:02.535786 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 10:33:02.535929 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":243,"threads_started":1}
I20260925 10:33:02.536942 8336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:02.658035 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 8206
I20260925 10:33:02.679927 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 8345
Sep 25 10:33:02 dist-test-slave-dxdd krb5kdc[7950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260925 10:33:02.693917 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:02.680613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.0.97.3:40741)
0925 10:33:02.681065 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:02.681069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:02.681086 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:33:02.681146 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:02.681150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:02.681208 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:02.681309 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:02.682588 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:02.683241 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:02.684407 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:02.684824 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:02.684916 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:02.688978 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:02.688991 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:02.689007 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:02.689045 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:02.691795 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:02.692347 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:02.692354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:02.692356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:02.692470 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:02.692753 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:02.692756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:02.692758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:02.693045 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:02.693317 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:02.693524 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 10:33:02.693674 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":259,"threads_started":1}
I20260925 10:33:02.694626 8475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:02.814493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 8345
I20260925 10:33:02.835600 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 7963
I20260925 10:33:02.836753 8035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:02.949772 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 7963
2026-09-25T10:33:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5339 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:33:03 dist-test-slave-dxdd krb5kdc[8605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:33:03 dist-test-slave-dxdd krb5kdc[8605](info): set up 2 sockets
Sep 25 10:33:03 dist-test-slave-dxdd krb5kdc[8605](info): commencing operation
krb5kdc: starting...
W20260925 10:33:04.035280 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:33:04 dist-test-slave-dxdd krb5kdc[8605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:33:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:33:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:33:04.233717 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41013
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:33:04.543206 8618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:04.543490 8618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:04.543577 8618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:04.550700 8618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:33:04.550797 8618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:04.550848 8618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:04.550909 8618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:33:04.550956 8618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:33:04.560405 8618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:04.561619 8618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:04.562917 8618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:04.574656 8624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:04.575256 8626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:04.575292 8623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:04.575620 8618 server_base.cc:1034] running on GCE node
I20260925 10:33:04.576447 8618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:04.578013 8618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:04.579278 8618 hybrid_clock.cc:648] HybridClock initialized: now 1790332384579213 us; error 50 us; skew 500 ppm
I20260925 10:33:04.579656 8618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T10:33:04Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1698 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:33:04 dist-test-slave-dxdd krb5kdc[8632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:33:04 dist-test-slave-dxdd krb5kdc[8632](info): set up 2 sockets
Sep 25 10:33:04 dist-test-slave-dxdd krb5kdc[8632](info): commencing operation
krb5kdc: starting...
W20260925 10:33:05.747308 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:33:05 dist-test-slave-dxdd krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:33:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:33:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:33:05.962267 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36691
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:33:06.282907 8645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:06.283165 8645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:06.283249 8645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:06.290123 8645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:33:06.290220 8645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:06.290264 8645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:06.290386 8645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:33:06.290438 8645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:33:06.299774 8645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:06.300966 8645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:06.302167 8645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:06.313664 8650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:06.313709 8651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:06.313664 8653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:06.320953 8645 server_base.cc:1034] running on GCE node
I20260925 10:33:06.321640 8645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:06.323000 8645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:06.324208 8645 hybrid_clock.cc:648] HybridClock initialized: now 1790332386324169 us; error 25 us; skew 500 ppm
I20260925 10:33:06.324560 8645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:33:06 dist-test-slave-dxdd krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:33:06.327977 8645 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:33:06.329577 8645 webserver.cc:460] Webserver started at http://127.0.97.62:33133/ using document root <none> and password file <none>
I20260925 10:33:06.330366 8645 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:06.330509 8645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:06.330826 8645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:06.333711 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "d43564bea34349f99777cb62edb77b4e"
format_stamp: "Formatted at 2026-09-25 10:33:06 on dist-test-slave-dxdd"
server_key: "d0fc61dbfe06ae709ad49affb322c3ff"
server_key_iv: "7fa1cae3fd73a47efc24b27b9be7264c"
server_key_version: "encryptionkey@0"
I20260925 10:33:06.334537 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "d43564bea34349f99777cb62edb77b4e"
format_stamp: "Formatted at 2026-09-25 10:33:06 on dist-test-slave-dxdd"
server_key: "d0fc61dbfe06ae709ad49affb322c3ff"
server_key_iv: "7fa1cae3fd73a47efc24b27b9be7264c"
server_key_version: "encryptionkey@0"
I20260925 10:33:06.339426 8645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 10:33:06.342669 8660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:06.343858 8645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:33:06.344026 8645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "d43564bea34349f99777cb62edb77b4e"
format_stamp: "Formatted at 2026-09-25 10:33:06 on dist-test-slave-dxdd"
server_key: "d0fc61dbfe06ae709ad49affb322c3ff"
server_key_iv: "7fa1cae3fd73a47efc24b27b9be7264c"
server_key_version: "encryptionkey@0"
I20260925 10:33:06.344208 8645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:06.359581 8645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:06.363337 8645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:06.363607 8645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:06.381410 8645 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36691
I20260925 10:33:06.381434 8711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36691 every 8 connection(s)
I20260925 10:33:06.382750 8645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:33:06.386243 8712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:06.388301 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 8645
I20260925 10:33:06.388497 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:33:06.389330 388 external_mini_cluster.cc:1428] Setting key fad64bf1d42c845ab0feb0d59908e9d5
I20260925 10:33:06.398967 8712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e: Bootstrap starting.
I20260925 10:33:06.402848 8712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e: Neither blocks nor log segments found. Creating new log.
Sep 25 10:33:06 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:33:06.404042 8712 log.cc:824] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:33:06.407444 8712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e: No bootstrap required, opened a new log
I20260925 10:33:06.413146 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:06.391384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.0.97.62:36691)
0925 10:33:06.392222 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:06.392233 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:06.392273 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:33:06.393959 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:06.393980 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:06.394419 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:06.394734 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:06.396464 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:06.397751 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:06.399543 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:06.400190 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:06.400317 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:06.404968 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:06.404975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:06.404992 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:06.405025 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:06.408787 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:06.409451 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:06.409461 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:06.409475 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:06.409665 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:06.409981 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:06.409986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:06.409990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:06.410740 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:06.411044 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:06.411336 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 10:33:06.411787 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":378,"threads_started":1}
I20260925 10:33:06.417363 8712 raft_consensus.cc:348] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43564bea34349f99777cb62edb77b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36691 } }
I20260925 10:33:06.417663 8712 raft_consensus.cc:374] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:06.417738 8712 raft_consensus.cc:729] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43564bea34349f99777cb62edb77b4e, State: Initialized, Role: FOLLOWER
I20260925 10:33:06.418188 8712 consensus_queue.cc:260] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43564bea34349f99777cb62edb77b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36691 } }
I20260925 10:33:06.418470 8712 raft_consensus.cc:388] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:33:06.418576 8712 raft_consensus.cc:482] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:33:06.418691 8712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:06.420782 8712 raft_consensus.cc:504] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43564bea34349f99777cb62edb77b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36691 } }
I20260925 10:33:06.421330 8712 leader_election.cc:302] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43564bea34349f99777cb62edb77b4e; no voters:
I20260925 10:33:06.421891 8712 leader_election.cc:288] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:33:06.422024 8717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:33:06.422755 8717 raft_consensus.cc:686] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 1 LEADER]: Becoming Leader. State: Replica: d43564bea34349f99777cb62edb77b4e, State: Running, Role: LEADER
I20260925 10:33:06.423270 8717 consensus_queue.cc:237] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43564bea34349f99777cb62edb77b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36691 } }
I20260925 10:33:06.423956 8712 sys_catalog.cc:564] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:33:06.425969 8718 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d43564bea34349f99777cb62edb77b4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43564bea34349f99777cb62edb77b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36691 } } }
I20260925 10:33:06.426015 8719 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d43564bea34349f99777cb62edb77b4e. Latest consensus state: current_term: 1 leader_uuid: "d43564bea34349f99777cb62edb77b4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43564bea34349f99777cb62edb77b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36691 } } }
I20260925 10:33:06.426410 8718 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [sys.catalog]: This master's current role is: LEADER
I20260925 10:33:06.426410 8719 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [sys.catalog]: This master's current role is: LEADER
I20260925 10:33:06.427979 8722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:33:06.432781 8722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:33:06.440908 8722 catalog_manager.cc:1347] Generated new cluster ID: 3762817772744724b64827875bcbdb37
I20260925 10:33:06.441010 8722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:33:06.484795 8722 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:33:06.486619 8722 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:33:06.504604 8722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e: Generated new TSK 0
I20260925 10:33:06.505412 8722 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:33:06.569705 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36691
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:33:06.895859 8738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:06.896126 8738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:06.896235 8738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:06.903195 8738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:06.903294 8738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:06.903400 8738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:33:06.912879 8738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:06.914037 8738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:06.915441 8738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:06.928035 8743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:06.928071 8746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:06.927975 8744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:06.928819 8738 server_base.cc:1034] running on GCE node
I20260925 10:33:06.929492 8738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:06.930478 8738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:06.931694 8738 hybrid_clock.cc:648] HybridClock initialized: now 1790332386931661 us; error 48 us; skew 500 ppm
I20260925 10:33:06.931998 8738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:33:06 dist-test-slave-dxdd krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:33:06.941437 8738 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:33:06.942895 8738 webserver.cc:460] Webserver started at http://127.0.97.1:38751/ using document root <none> and password file <none>
I20260925 10:33:06.943545 8738 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:06.943641 8738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:06.943898 8738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:06.947064 8738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "5cb3128247a44da7be3a0d4f5dba3c22"
format_stamp: "Formatted at 2026-09-25 10:33:06 on dist-test-slave-dxdd"
server_key: "5133be2d973beb0dedcf95baa038f5d9"
server_key_iv: "1c0d960bd270c19a27952c0f7b6cd6e7"
server_key_version: "encryptionkey@0"
I20260925 10:33:06.947903 8738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5cb3128247a44da7be3a0d4f5dba3c22"
format_stamp: "Formatted at 2026-09-25 10:33:06 on dist-test-slave-dxdd"
server_key: "5133be2d973beb0dedcf95baa038f5d9"
server_key_iv: "1c0d960bd270c19a27952c0f7b6cd6e7"
server_key_version: "encryptionkey@0"
I20260925 10:33:06.953056 8738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:33:06.956318 8753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:06.957556 8738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:33:06.957680 8738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "5cb3128247a44da7be3a0d4f5dba3c22"
format_stamp: "Formatted at 2026-09-25 10:33:06 on dist-test-slave-dxdd"
server_key: "5133be2d973beb0dedcf95baa038f5d9"
server_key_iv: "1c0d960bd270c19a27952c0f7b6cd6e7"
server_key_version: "encryptionkey@0"
I20260925 10:33:06.957832 8738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:06.973279 8738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:06.977255 8738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:06.977468 8738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:06.978259 8738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:33:06.979508 8738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:06.979578 8738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:06.979660 8738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:06.979714 8738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:07.011379 8738 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34709
I20260925 10:33:07.011425 8865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34709 every 8 connection(s)
I20260925 10:33:07.012562 8738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:33:07.021251 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 8738
I20260925 10:33:07.021479 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:33:07.022241 388 external_mini_cluster.cc:1428] Setting key 7b199407bd11c127c7e5bf908a12dff3
Sep 25 10:33:07 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:33:07.033388 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:07.015052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36189 (local address 127.0.97.62:36691)
0925 10:33:07.015475 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:07.015479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:07.016416 (+ 937us) server_negotiation.cc:407] Connection header received
0925 10:33:07.018110 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:07.018115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:07.018205 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:07.018341 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:07.019750 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:07.020543 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:07.022988 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:07.023522 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:07.023653 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:07.027521 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:07.027526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:07.027529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:07.027557 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:07.030370 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:07.031060 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:07.031066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:07.031071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:07.031223 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:07.031603 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:07.031611 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:07.031617 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:07.031965 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:07.032107 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:07.032695 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 10:33:07.032922 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":236,"threads_started":1}
I20260925 10:33:07.033947 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:07.015607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:36189)
0925 10:33:07.016266 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:07.016305 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:07.017857 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:07.018549 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:07.018557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:07.019049 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:07.019595 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:07.019610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:07.020722 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:07.020726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:07.022820 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:07.022833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:07.023755 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:07.023760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:07.024001 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:07.024769 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:07.024798 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:07.027322 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:07.030503 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:07.030509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:07.030520 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:07.030935 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:07.031327 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:07.031330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:07.031332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:07.031490 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:07.032077 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:07.032082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:07.032350 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 10:33:07.032545 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 10:33:07.032882 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":269,"threads_started":1}
I20260925 10:33:07.035424 8866 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36691
I20260925 10:33:07.035717 8866 heartbeater.cc:464] Registering TS with master...
I20260925 10:33:07.036581 8866 heartbeater.cc:511] Master 127.0.97.62:36691 requested a full tablet report, sending...
I20260925 10:33:07.038655 8677 ts_manager.cc:194] Registered new tserver with Master: 5cb3128247a44da7be3a0d4f5dba3c22 (127.0.97.1:34709)
I20260925 10:33:07.040131 8677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36189
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:33:07.072708 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36691
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:33:07.385520 8872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:07.385855 8872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:07.385999 8872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:07.397325 8872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:07.397467 8872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:07.397652 8872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:33:07.408711 8872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:07.409832 8872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:07.411181 8872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:07.424736 8880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:07.424773 8878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:07.424697 8877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:07.425478 8872 server_base.cc:1034] running on GCE node
I20260925 10:33:07.426087 8872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:07.427042 8872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:07.428284 8872 hybrid_clock.cc:648] HybridClock initialized: now 1790332387428218 us; error 54 us; skew 500 ppm
I20260925 10:33:07.428639 8872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:33:07 dist-test-slave-dxdd krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:33:07.431740 8872 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:33:07.433112 8872 webserver.cc:460] Webserver started at http://127.0.97.2:42383/ using document root <none> and password file <none>
I20260925 10:33:07.433808 8872 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:07.433952 8872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:07.434247 8872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:07.437031 8872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "dd2efee9c72a4abc8c1ba1913f417219"
format_stamp: "Formatted at 2026-09-25 10:33:07 on dist-test-slave-dxdd"
server_key: "a9e78a79276d5cc0590679c36d690744"
server_key_iv: "d131db691438a79c3ba6ff518733c0ff"
server_key_version: "encryptionkey@0"
I20260925 10:33:07.437816 8872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "dd2efee9c72a4abc8c1ba1913f417219"
format_stamp: "Formatted at 2026-09-25 10:33:07 on dist-test-slave-dxdd"
server_key: "a9e78a79276d5cc0590679c36d690744"
server_key_iv: "d131db691438a79c3ba6ff518733c0ff"
server_key_version: "encryptionkey@0"
I20260925 10:33:07.442826 8872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 10:33:07.446096 8887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:07.447399 8872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:33:07.447568 8872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "dd2efee9c72a4abc8c1ba1913f417219"
format_stamp: "Formatted at 2026-09-25 10:33:07 on dist-test-slave-dxdd"
server_key: "a9e78a79276d5cc0590679c36d690744"
server_key_iv: "d131db691438a79c3ba6ff518733c0ff"
server_key_version: "encryptionkey@0"
I20260925 10:33:07.447829 8872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:07.469514 8872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:07.473850 8872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:07.474126 8872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:07.475108 8872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:33:07.476344 8872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:07.476454 8872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:07.476599 8872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:07.476670 8872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:07.506933 8872 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38523
I20260925 10:33:07.506985 8999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38523 every 8 connection(s)
I20260925 10:33:07.508175 8872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:33:07.511639 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 8872
I20260925 10:33:07.511809 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:33:07.512372 388 external_mini_cluster.cc:1428] Setting key 83cda0530d4776ea732c53e947432d6e
Sep 25 10:33:07 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:33:07.530965 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:07.510676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53225 (local address 127.0.97.62:36691)
0925 10:33:07.510841 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:07.510846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:07.512179 (+ 1333us) server_negotiation.cc:407] Connection header received
0925 10:33:07.514414 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:07.514418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:07.514480 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:07.514563 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:07.516643 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:07.517325 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:07.519735 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:07.520148 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:07.520256 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:07.523892 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:07.523900 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:07.523908 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:07.523951 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:07.527504 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:07.528165 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:07.528173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:07.528179 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:07.528331 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:07.528929 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:07.528936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:07.528942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:07.529310 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:07.529468 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:07.530219 (+ 751us) server_negotiation.cc:299] Negotiation successful
0925 10:33:07.530511 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:33:07.531669 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:07.511161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:53225)
0925 10:33:07.512003 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:07.512056 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:07.514101 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:07.514834 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:07.514848 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:07.515653 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:07.516464 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:07.516490 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:07.517475 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:07.517483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:07.519578 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:07.519595 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:07.520372 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:07.520379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:07.520690 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:07.521521 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:07.521548 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:07.523646 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:07.527627 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:07.527634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:07.527646 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:07.528038 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:07.528484 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:07.528526 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:07.528532 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:07.528745 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:07.529427 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:07.529432 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:07.529809 (+ 377us) client_negotiation.cc:769] Sending connection context
0925 10:33:07.530084 (+ 275us) client_negotiation.cc:240] Negotiation successful
0925 10:33:07.530525 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":292,"threads_started":1}
I20260925 10:33:07.533160 9000 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36691
I20260925 10:33:07.533481 9000 heartbeater.cc:464] Registering TS with master...
I20260925 10:33:07.534185 9000 heartbeater.cc:511] Master 127.0.97.62:36691 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260925 10:33:07.535697 8677 ts_manager.cc:194] Registered new tserver with Master: dd2efee9c72a4abc8c1ba1913f417219 (127.0.97.2:38523)
I20260925 10:33:07.536549 8677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:33:07.562031 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36691
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:33:07.874186 9005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:07.874467 9005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:07.874558 9005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:07.881492 9005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:07.881590 9005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:07.881696 9005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:33:07.890908 9005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:07.892112 9005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:07.893291 9005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:07.905893 9010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:07.905820 9013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:07.905784 9011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:07.907016 9005 server_base.cc:1034] running on GCE node
I20260925 10:33:07.907810 9005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:07.908682 9005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:07.909916 9005 hybrid_clock.cc:648] HybridClock initialized: now 1790332387909870 us; error 63 us; skew 500 ppm
I20260925 10:33:07.910228 9005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:33:07 dist-test-slave-dxdd krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:33:07.913561 9005 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:33:07.914970 9005 webserver.cc:460] Webserver started at http://127.0.97.3:46171/ using document root <none> and password file <none>
I20260925 10:33:07.915614 9005 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:07.915753 9005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:07.916057 9005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:07.918977 9005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "cdc15dd865c94a2fbb7c1c26d98a52d9"
format_stamp: "Formatted at 2026-09-25 10:33:07 on dist-test-slave-dxdd"
server_key: "d81b54b4852a259469c98532204343e9"
server_key_iv: "312e9446cb554b45101d49065982e4bc"
server_key_version: "encryptionkey@0"
I20260925 10:33:07.919709 9005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cdc15dd865c94a2fbb7c1c26d98a52d9"
format_stamp: "Formatted at 2026-09-25 10:33:07 on dist-test-slave-dxdd"
server_key: "d81b54b4852a259469c98532204343e9"
server_key_iv: "312e9446cb554b45101d49065982e4bc"
server_key_version: "encryptionkey@0"
I20260925 10:33:07.924597 9005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:33:07.927846 9020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:07.929075 9005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:33:07.929235 9005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "cdc15dd865c94a2fbb7c1c26d98a52d9"
format_stamp: "Formatted at 2026-09-25 10:33:07 on dist-test-slave-dxdd"
server_key: "d81b54b4852a259469c98532204343e9"
server_key_iv: "312e9446cb554b45101d49065982e4bc"
server_key_version: "encryptionkey@0"
I20260925 10:33:07.929442 9005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:07.944766 9005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:07.948452 9005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:07.948720 9005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:07.949579 9005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:33:07.950882 9005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:07.950978 9005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:07.951121 9005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:07.951169 9005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:07.983172 9005 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33745
I20260925 10:33:07.983213 9132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33745 every 8 connection(s)
I20260925 10:33:07.984442 9005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:33:07.989676 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 9005
I20260925 10:33:07.989835 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:33:07.990481 388 external_mini_cluster.cc:1428] Setting key f2317e9eaf000fbe43e3af180a6969c3
Sep 25 10:33:07 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:33:08.003021 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:07.986927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48137 (local address 127.0.97.62:36691)
0925 10:33:07.987102 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:07.987108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:07.988233 (+ 1125us) server_negotiation.cc:407] Connection header received
0925 10:33:07.990183 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:07.990187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:07.990249 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:07.990409 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:07.992332 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:07.993142 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:07.994768 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:07.995141 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:07.995233 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:07.998199 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:07.998204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:07.998207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:07.998234 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:08.000547 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:08.001163 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:08.001167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:08.001169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:08.001285 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:08.001626 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:08.001629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:08.001631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:08.001881 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:08.001984 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:08.002599 (+ 615us) server_negotiation.cc:299] Negotiation successful
0925 10:33:08.002741 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:33:08.003999 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:07.987404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:48137)
0925 10:33:07.988067 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:07.988101 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:07.989895 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:07.990651 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:07.990663 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:07.991351 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:07.992149 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:07.992167 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:07.993323 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:07.993329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:07.994624 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:07.994635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:07.995311 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:07.995315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:07.995490 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:07.996063 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:07.996087 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:07.998037 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:08.000671 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:08.000678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:08.000688 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:08.001067 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:08.001375 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:08.001378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:08.001380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:08.001538 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:08.001987 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:08.001992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:08.002250 (+ 258us) client_negotiation.cc:769] Sending connection context
0925 10:33:08.002511 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 10:33:08.002834 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":259,"threads_started":1}
I20260925 10:33:08.005437 9133 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36691
I20260925 10:33:08.005683 9133 heartbeater.cc:464] Registering TS with master...
I20260925 10:33:08.006361 9133 heartbeater.cc:511] Master 127.0.97.62:36691 requested a full tablet report, sending...
I20260925 10:33:08.007599 8677 ts_manager.cc:194] Registered new tserver with Master: cdc15dd865c94a2fbb7c1c26d98a52d9 (127.0.97.3:33745)
I20260925 10:33:08.008347 8677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48137
I20260925 10:33:08.017379 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
I20260925 10:33:08.043608 8866 heartbeater.cc:503] Master 127.0.97.62:36691 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260925 10:33:08.141547 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 8738
Sep 25 10:33:08 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:33:08.157404 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:08.142657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.0.97.1:34709)
0925 10:33:08.143267 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:08.143274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:08.143302 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:33:08.143383 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:08.143389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:08.143613 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:08.143741 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:08.145066 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:08.146046 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:08.147372 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:08.148042 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:08.148168 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:08.152184 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:08.152202 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:08.152222 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:08.152263 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:08.155120 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:08.155690 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:08.155696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:08.155699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:08.155807 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:08.156104 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:08.156107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:08.156109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:08.156406 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:08.156621 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:08.156812 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 10:33:08.156987 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":308,"threads_started":1}
I20260925 10:33:08.158742 8861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:08.240193 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 8738
I20260925 10:33:08.256726 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 8872
Sep 25 10:33:08 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:33:08.271571 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:08.257602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42902 (local address 127.0.97.2:38523)
0925 10:33:08.258110 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:08.258116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:08.258139 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:33:08.258196 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:08.258202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:08.258493 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:08.258666 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:08.259855 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:08.260839 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:08.262043 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:08.262712 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:08.262842 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:08.266766 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:08.266776 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:08.266790 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:08.266826 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:08.269363 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:08.269901 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:08.269907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:08.269909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:08.270029 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:08.270364 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:08.270368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:08.270370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:08.270668 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:08.270878 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:08.271115 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 10:33:08.271292 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":255,"threads_started":1}
I20260925 10:33:08.272746 8995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:08.354167 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 8872
I20260925 10:33:08.369956 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 9005
Sep 25 10:33:08 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:33:08.385950 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:08.371058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.0.97.3:33745)
0925 10:33:08.371639 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:08.371645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:08.371673 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:33:08.371731 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:08.371736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:08.371929 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:08.372115 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:08.373361 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:08.374423 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:08.375639 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:08.376198 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:08.376333 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:08.380395 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:08.380405 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:08.380419 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:08.380448 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:08.383312 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:08.383958 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:08.383965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:08.383967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:08.384090 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:08.384438 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:08.384444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:08.384448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:08.384765 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:08.385085 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:08.385366 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 10:33:08.385632 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":292,"threads_started":1}
I20260925 10:33:08.387285 9128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:08.467948 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 9005
I20260925 10:33:08.483214 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 8645
I20260925 10:33:08.484472 8707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:08.584600 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 8645
I20260925 10:33:08.604926 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36691
--webserver_interface=127.0.97.62
--webserver_port=33133
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:33:08.916855 9150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:08.917196 9150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:08.917371 9150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:08.924512 9150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:33:08.924647 9150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:08.924764 9150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:08.924857 9150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:33:08.924943 9150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:33:08.934535 9150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:08.935897 9150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:08.937222 9150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:08.948602 9158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:08.948683 9156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:08.948650 9155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:08.949723 9150 server_base.cc:1034] running on GCE node
I20260925 10:33:08.950587 9150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:08.951893 9150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:08.953122 9150 hybrid_clock.cc:648] HybridClock initialized: now 1790332388953079 us; error 53 us; skew 500 ppm
I20260925 10:33:08.953507 9150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:33:08 dist-test-slave-dxdd krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:33:08.956526 9150 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:33:08.957872 9150 webserver.cc:460] Webserver started at http://127.0.97.62:33133/ using document root <none> and password file <none>
I20260925 10:33:08.958568 9150 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:08.958711 9150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:08.963693 9150 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:33:08.966219 9165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:08.967434 9150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 10:33:08.967607 9150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "d43564bea34349f99777cb62edb77b4e"
format_stamp: "Formatted at 2026-09-25 10:33:06 on dist-test-slave-dxdd"
server_key: "d0fc61dbfe06ae709ad49affb322c3ff"
server_key_iv: "7fa1cae3fd73a47efc24b27b9be7264c"
server_key_version: "encryptionkey@0"
I20260925 10:33:08.968482 9150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:08.992678 9150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:08.996984 9150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:08.997238 9150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:09.014839 9150 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36691
I20260925 10:33:09.014875 9216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36691 every 8 connection(s)
I20260925 10:33:09.016229 9150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:33:09.022850 9217 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:33:09.024390 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 9150
I20260925 10:33:09.025141 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:34709
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:36691
--webserver_port=38751
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:33:09.026135 9217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e: Bootstrap starting.
I20260925 10:33:09.050673 9217 log.cc:824] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:33:09.061684 9217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:33:09.062283 9217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e: Bootstrap complete.
I20260925 10:33:09.075074 9217 raft_consensus.cc:348] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43564bea34349f99777cb62edb77b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36691 } }
I20260925 10:33:09.076159 9217 raft_consensus.cc:729] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43564bea34349f99777cb62edb77b4e, State: Initialized, Role: FOLLOWER
I20260925 10:33:09.076817 9217 consensus_queue.cc:260] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43564bea34349f99777cb62edb77b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36691 } }
I20260925 10:33:09.077106 9217 raft_consensus.cc:388] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:33:09.077265 9217 raft_consensus.cc:482] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:33:09.077469 9217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:33:09.082929 9217 raft_consensus.cc:504] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43564bea34349f99777cb62edb77b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36691 } }
I20260925 10:33:09.083454 9217 leader_election.cc:302] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43564bea34349f99777cb62edb77b4e; no voters:
I20260925 10:33:09.084110 9217 leader_election.cc:288] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:33:09.084254 9221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:33:09.085558 9221 raft_consensus.cc:686] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [term 2 LEADER]: Becoming Leader. State: Replica: d43564bea34349f99777cb62edb77b4e, State: Running, Role: LEADER
I20260925 10:33:09.086088 9217 sys_catalog.cc:564] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:33:09.086053 9221 consensus_queue.cc:237] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43564bea34349f99777cb62edb77b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36691 } }
I20260925 10:33:09.088553 9223 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d43564bea34349f99777cb62edb77b4e. Latest consensus state: current_term: 2 leader_uuid: "d43564bea34349f99777cb62edb77b4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43564bea34349f99777cb62edb77b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36691 } } }
I20260925 10:33:09.088524 9222 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d43564bea34349f99777cb62edb77b4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43564bea34349f99777cb62edb77b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36691 } } }
I20260925 10:33:09.088948 9223 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [sys.catalog]: This master's current role is: LEADER
I20260925 10:33:09.088949 9222 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e [sys.catalog]: This master's current role is: LEADER
I20260925 10:33:09.101377 9227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:33:09.106891 9227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:33:09.108196 9227 catalog_manager.cc:1259] Loaded cluster ID: 3762817772744724b64827875bcbdb37
I20260925 10:33:09.108289 9227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:33:09.112533 9227 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:33:09.113823 9227 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d43564bea34349f99777cb62edb77b4e: Loaded TSK: 0
I20260925 10:33:09.119067 9227 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:33:09.374571 9219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:09.374881 9219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:09.375034 9219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:09.382092 9219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:09.382241 9219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:09.382436 9219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:33:09.392093 9219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:34709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38751
--tserver_master_addrs=127.0.97.62:36691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:09.393406 9219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:09.394758 9219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:09.409268 9243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:09.409297 9244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:09.409821 9246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:09.410342 9219 server_base.cc:1034] running on GCE node
I20260925 10:33:09.411011 9219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:09.412103 9219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:09.413334 9219 hybrid_clock.cc:648] HybridClock initialized: now 1790332389413265 us; error 61 us; skew 500 ppm
I20260925 10:33:09.413687 9219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:33:09 dist-test-slave-dxdd krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332389, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:33:09.417608 9219 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260925 10:33:09.419348 9219 webserver.cc:460] Webserver started at http://127.0.97.1:38751/ using document root <none> and password file <none>
I20260925 10:33:09.420200 9219 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:09.420339 9219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:09.427271 9219 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:33:09.430450 9253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:09.432086 9219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:33:09.432245 9219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "5cb3128247a44da7be3a0d4f5dba3c22"
format_stamp: "Formatted at 2026-09-25 10:33:06 on dist-test-slave-dxdd"
server_key: "5133be2d973beb0dedcf95baa038f5d9"
server_key_iv: "1c0d960bd270c19a27952c0f7b6cd6e7"
server_key_version: "encryptionkey@0"
I20260925 10:33:09.433578 9219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:09.474388 9219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:09.478250 9219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:09.478528 9219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:09.479458 9219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:33:09.480680 9219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:09.480772 9219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:09.480852 9219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:09.480909 9219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:09.513162 9219 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34709
I20260925 10:33:09.513218 9365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34709 every 8 connection(s)
I20260925 10:33:09.514698 9219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:33:09.520008 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 9219
I20260925 10:33:09.520584 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38523
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:36691
--webserver_port=42383
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:33:09 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:09 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:33:09.535709 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:09.517727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44413 (local address 127.0.97.62:36691)
0925 10:33:09.518416 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:09.518427 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:09.518757 (+ 330us) server_negotiation.cc:407] Connection header received
0925 10:33:09.521108 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:09.521130 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:09.521608 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:09.522003 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:09.523883 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.525844 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.527428 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.528572 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.528771 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:09.534856 (+ 6085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44413: BlockingRecv error: recv got EOF from 127.0.97.1:44413 (error 108)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":244,"threads_started":1}
I20260925 10:33:09.535758 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:09.517734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:44413)
0925 10:33:09.518587 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:09.518638 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:09.520482 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:09.522129 (+ 1647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:09.522136 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:09.522944 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:09.523684 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:09.523702 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.526009 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.526013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:09.527277 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:09.527288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.528798 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.528807 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:09.529124 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:09.529826 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:09.529859 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:09.534570 (+ 4711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":356,"threads_started":1}
W20260925 10:33:09.536571 9366 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36691 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:36691: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 25 10:33:09 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:09 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:09.546221 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:09.537207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:60793)
0925 10:33:09.537384 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:09.537405 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:09.537508 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:09.537882 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:09.537885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:09.538166 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:09.538353 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:09.538365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.539487 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.539496 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:09.541266 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:09.541283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.541929 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.541938 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:09.542189 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:09.542832 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:09.542861 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:09.546012 (+ 3151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 10:33:09.546512 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:09.537381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60793 (local address 127.0.97.62:36691)
0925 10:33:09.537582 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:09.537590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:09.537611 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:33:09.537667 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:09.537675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:09.537753 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:09.537855 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:09.538530 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.539336 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.541401 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.541783 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.541872 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:09.546265 (+ 4393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60793: BlockingRecv error: recv got EOF from 127.0.97.1:60793 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 10:33:09 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:09 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:09.554603 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:09.547006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:51973)
0925 10:33:09.547180 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:09.547199 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:09.547296 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:09.547701 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:09.547708 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:09.548136 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:09.548401 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:09.548416 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.549443 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.549450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:09.550426 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:09.550436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.551112 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.551117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:09.551293 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:09.551855 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:09.551871 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:09.554406 (+ 2535us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 10:33:09.554855 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:09.547181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51973 (local address 127.0.97.62:36691)
0925 10:33:09.547348 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:09.547353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:09.547371 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:33:09.547439 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:09.547444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:09.547520 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:09.547609 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:09.548586 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.549306 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.550559 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.550974 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.551103 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:09.554613 (+ 3510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51973: BlockingRecv error: recv got EOF from 127.0.97.1:51973 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 10:33:09.554971 9366 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:33:09.847220 9370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:09.847486 9370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:09.847666 9370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:09.854822 9370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:09.854957 9370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:09.855149 9370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:33:09.864738 9370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42383
--tserver_master_addrs=127.0.97.62:36691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:09.866022 9370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:09.867300 9370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:09.880079 9375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:09.880039 9378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:09.880124 9376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:09.883669 9370 server_base.cc:1034] running on GCE node
I20260925 10:33:09.884243 9370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:09.885165 9370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:09.886420 9370 hybrid_clock.cc:648] HybridClock initialized: now 1790332389886376 us; error 48 us; skew 500 ppm
I20260925 10:33:09.886729 9370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:33:09 dist-test-slave-dxdd krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332389, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:33:09.889688 9370 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260925 10:33:09.891147 9370 webserver.cc:460] Webserver started at http://127.0.97.2:42383/ using document root <none> and password file <none>
I20260925 10:33:09.891839 9370 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:09.892063 9370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:09.897241 9370 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 10:33:09.899797 9385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:09.900985 9370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:33:09.901157 9370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "dd2efee9c72a4abc8c1ba1913f417219"
format_stamp: "Formatted at 2026-09-25 10:33:07 on dist-test-slave-dxdd"
server_key: "a9e78a79276d5cc0590679c36d690744"
server_key_iv: "d131db691438a79c3ba6ff518733c0ff"
server_key_version: "encryptionkey@0"
I20260925 10:33:09.902087 9370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:09.919216 9370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:09.923583 9370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:09.923849 9370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:09.924777 9370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:33:09.926116 9370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:09.926229 9370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:09.926374 9370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:09.926460 9370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:09.959337 9370 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38523
I20260925 10:33:09.959424 9497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38523 every 8 connection(s)
I20260925 10:33:09.960777 9370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:33:09.965737 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 9370
I20260925 10:33:09.966374 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33745
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:36691
--webserver_port=46171
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:33:09 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:09 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:33:09.976652 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:09.963314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47789 (local address 127.0.97.62:36691)
0925 10:33:09.963489 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:09.963494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:09.964636 (+ 1142us) server_negotiation.cc:407] Connection header received
0925 10:33:09.966620 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:09.966637 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:09.966701 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:09.966758 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:09.968373 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.969059 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.970760 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.971177 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.971333 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:09.976404 (+ 5071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47789: BlockingRecv error: recv got EOF from 127.0.97.2:47789 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:33:09.977382 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:09.963778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:47789)
0925 10:33:09.964467 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:09.964511 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:09.966242 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:09.967145 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:09.967153 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:09.967659 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:09.968193 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:09.968210 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.969241 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.969251 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:09.970620 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:09.970638 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.971416 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.971423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:09.971673 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:09.972393 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:09.972421 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:09.976234 (+ 3813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":274,"threads_started":1}
W20260925 10:33:09.978042 9498 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36691 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:36691: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 25 10:33:09 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:09 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:09.986147 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:09.978705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:59557)
0925 10:33:09.978880 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:09.978899 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:09.979008 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:09.979331 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:09.979336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:09.979650 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:09.979776 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:09.979784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.980735 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.980742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:09.981611 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:09.981620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.982292 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.982297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:09.982484 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:09.983053 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:09.983069 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:09.985851 (+ 2782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 10:33:09.986513 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:09.978768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59557 (local address 127.0.97.62:36691)
0925 10:33:09.978930 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:09.978938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:09.978956 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:33:09.979132 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:09.979137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:09.979207 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:09.979311 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:09.979905 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.980592 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.981745 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.982144 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.982282 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:09.986163 (+ 3881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59557: BlockingRecv error: recv got EOF from 127.0.97.2:59557 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 10:33:09 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:09 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:09.994653 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:09.986911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:34977)
0925 10:33:09.987103 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:09.987122 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:09.987215 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:09.987520 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:09.987523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:09.987824 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:09.987956 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:09.987963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.989305 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.989308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:09.990201 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:09.990211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.991081 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.991085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:09.991203 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:09.991796 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:09.991813 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:09.994429 (+ 2616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260925 10:33:09.994858 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:09.987008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34977 (local address 127.0.97.62:36691)
0925 10:33:09.987201 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:09.987220 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:09.987242 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:33:09.987300 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:09.987307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:09.987390 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:09.987508 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:09.988091 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.989169 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.990345 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:09.990958 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:09.991077 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:09.994656 (+ 3579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34977: BlockingRecv error: recv got EOF from 127.0.97.2:34977 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 10:33:09.995000 9498 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:33:10.285903 9501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:10.286172 9501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:10.286264 9501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:10.293509 9501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:10.293609 9501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:10.293717 9501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:33:10.302989 9501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=46171
--tserver_master_addrs=127.0.97.62:36691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:10.304157 9501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:10.305311 9501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:10.317803 9506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:10.317749 9507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:10.317764 9509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:10.318610 9501 server_base.cc:1034] running on GCE node
I20260925 10:33:10.319151 9501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:10.319990 9501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:10.321218 9501 hybrid_clock.cc:648] HybridClock initialized: now 1790332390321184 us; error 64 us; skew 500 ppm
I20260925 10:33:10.321585 9501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:33:10 dist-test-slave-dxdd krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:33:10.324486 9501 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260925 10:33:10.325794 9501 webserver.cc:460] Webserver started at http://127.0.97.3:46171/ using document root <none> and password file <none>
I20260925 10:33:10.326571 9501 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:10.326751 9501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:10.331573 9501 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 10:33:10.334054 9516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:10.335515 9501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:33:10.335712 9501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "cdc15dd865c94a2fbb7c1c26d98a52d9"
format_stamp: "Formatted at 2026-09-25 10:33:07 on dist-test-slave-dxdd"
server_key: "d81b54b4852a259469c98532204343e9"
server_key_iv: "312e9446cb554b45101d49065982e4bc"
server_key_version: "encryptionkey@0"
I20260925 10:33:10.336642 9501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:10.352253 9501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:10.356561 9501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:10.356854 9501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:10.357788 9501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:33:10.359124 9501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:10.359251 9501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:10.359371 9501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:10.359457 9501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:10.391182 9501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33745
I20260925 10:33:10.391239 9628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33745 every 8 connection(s)
I20260925 10:33:10.392627 9501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:33:10.396986 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 9501
Sep 25 10:33:10 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:10 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:33:10.409827 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:10.395172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55365 (local address 127.0.97.62:36691)
0925 10:33:10.395359 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:10.395364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:10.396477 (+ 1113us) server_negotiation.cc:407] Connection header received
0925 10:33:10.398677 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:10.398686 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:10.398817 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:10.398944 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:10.400569 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.401380 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.403239 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.403852 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.403942 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:10.409542 (+ 5600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55365: BlockingRecv error: recv got EOF from 127.0.97.3:55365 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:33:10.410607 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:10.395635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:55365)
0925 10:33:10.396310 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:10.396347 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:10.398350 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:10.399218 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:10.399229 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:10.399836 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:10.400390 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:10.400406 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.401538 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.401543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:10.403110 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:10.403122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.404066 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.404076 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:10.404343 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:10.405060 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:10.405099 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:10.409364 (+ 4265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":260,"threads_started":1}
W20260925 10:33:10.411321 9629 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36691 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:36691: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260925 10:33:10.413959 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:10.397921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.0.97.62:36691)
0925 10:33:10.399084 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:10.399089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:10.399114 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:33:10.399162 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:10.399166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:10.399229 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:10.399337 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:10.400815 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.401763 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.403015 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.403518 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.403646 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:10.407069 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:10.407081 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:10.407120 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:10.407157 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:10.410184 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:10.411132 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:10.411139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:10.411144 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:10.411340 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:10.411749 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:10.411753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:10.411755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:10.412371 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:10.412614 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:10.413108 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 10:33:10.413525 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046,"thread_start_us":883,"threads_started":1}
Sep 25 10:33:10 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:10 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:10.419925 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:10.411875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:59869)
0925 10:33:10.412003 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:10.412019 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:10.412103 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:10.412347 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:10.412350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:10.412726 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:10.412941 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:10.412952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.414387 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.414395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:10.415487 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:10.415502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.416301 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.416310 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:10.416456 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:10.416984 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:10.417000 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:10.419748 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 10:33:10.420096 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:10.411950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59869 (local address 127.0.97.62:36691)
0925 10:33:10.412101 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:10.412106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:10.412122 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:33:10.412172 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:10.412176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:10.412237 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:10.412340 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:10.413090 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.414231 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.415616 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.416119 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.416242 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:10.419907 (+ 3665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59869: BlockingRecv error: recv got EOF from 127.0.97.3:59869 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 10:33:10 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:10 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:10.427484 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:10.420663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:51411)
0925 10:33:10.420803 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:10.420817 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:10.420923 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:10.421214 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:10.421218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:10.421504 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:10.421669 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:10.421682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.422702 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.422705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:10.423607 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:10.423616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.424170 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.424174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:10.424283 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:10.424782 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:10.424797 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:10.427302 (+ 2505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 10:33:10.427695 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:10.420756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51411 (local address 127.0.97.62:36691)
0925 10:33:10.420983 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:10.420989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:10.421006 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:33:10.421051 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:10.421055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:10.421101 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:10.421192 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:10.421808 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.422595 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.423721 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.424065 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.424148 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:10.427463 (+ 3315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51411: BlockingRecv error: recv got EOF from 127.0.97.3:51411 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
W20260925 10:33:10.428077 9629 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 10:33:10 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:10 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:10.563810 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:10.555574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:51747)
0925 10:33:10.556097 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:10.556132 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:10.556241 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:10.556549 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:10.556553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:10.557048 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:10.557236 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:10.557244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.558277 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.558282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:10.559242 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:10.559253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.560044 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.560050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:10.560246 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:10.560867 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:10.560884 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:10.563498 (+ 2614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":290,"threads_started":1}
W20260925 10:33:10.563905 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:10.555605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51747 (local address 127.0.97.62:36691)
0925 10:33:10.555811 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:10.555815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:10.556208 (+ 393us) server_negotiation.cc:407] Connection header received
0925 10:33:10.556334 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:10.556338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:10.556396 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:10.556479 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:10.557384 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.558126 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.559379 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.559846 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.559981 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:10.563679 (+ 3698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51747: BlockingRecv error: recv got EOF from 127.0.97.1:51747 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 10:33:10 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:10 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:11.003392 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:10.995496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:38133)
0925 10:33:10.995974 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:10.995990 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:10.996139 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:10.996409 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:10.996412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:10.996829 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:10.997045 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:10.997053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.997931 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.997934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:10.998901 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:10.998911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.999536 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.999540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:10.999647 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:11.000260 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:11.000277 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:11.003070 (+ 2793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":238,"threads_started":1}
W20260925 10:33:11.003453 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:10.995582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38133 (local address 127.0.97.62:36691)
0925 10:33:10.995840 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:10.995846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:10.996085 (+ 239us) server_negotiation.cc:407] Connection header received
0925 10:33:10.996233 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:10.996236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:10.996292 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:10.996369 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:10.997156 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.997823 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.999023 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:10.999400 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:10.999487 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:11.003237 (+ 3750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38133: BlockingRecv error: recv got EOF from 127.0.97.2:38133 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 25 10:33:11 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:11 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:11.436657 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:11.428501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:33033)
0925 10:33:11.429182 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:11.429197 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:11.429297 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:11.429632 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:11.429635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:11.430088 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:11.430291 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:11.430318 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:11.431270 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:11.431274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:11.432260 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:11.432270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:11.432881 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:11.432886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:11.432991 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:11.433582 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:11.433597 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:11.436337 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":396,"threads_started":1}
W20260925 10:33:11.436744 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:11.428573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33033 (local address 127.0.97.62:36691)
0925 10:33:11.428735 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:11.428740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:11.429272 (+ 532us) server_negotiation.cc:407] Connection header received
0925 10:33:11.429398 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:11.429403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:11.429490 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:11.429592 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:11.430436 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:11.431149 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:11.432379 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:11.432751 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:11.432836 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:11.436513 (+ 3677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33033: BlockingRecv error: recv got EOF from 127.0.97.3:33033 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 10:33:11 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:11 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:11.572808 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:11.564676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:55177)
0925 10:33:11.565225 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:11.565240 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:11.565348 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:11.565617 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:11.565620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:11.566000 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:11.566145 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:11.566153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:11.567133 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:11.567136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:11.568021 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:11.568031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:11.568653 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:11.568657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:11.568759 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:11.569311 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:11.569327 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:11.572503 (+ 3176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":330,"threads_started":1}
W20260925 10:33:11.573038 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:11.564765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55177 (local address 127.0.97.62:36691)
0925 10:33:11.564956 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:11.564960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:11.565314 (+ 354us) server_negotiation.cc:407] Connection header received
0925 10:33:11.565428 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:11.565431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:11.565498 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:11.565578 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:11.566258 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:11.567018 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:11.568134 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:11.568529 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:11.568622 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:11.572758 (+ 4136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55177: BlockingRecv error: recv got EOF from 127.0.97.1:55177 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 10:33:12 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:12 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:12.015974 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:12.004181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:34289)
0925 10:33:12.004662 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:12.004677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:12.004819 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:12.005128 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:12.005132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:12.005518 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:12.005666 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:12.005674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:12.006679 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:12.006685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:12.007579 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:12.007589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:12.008283 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:12.008290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:12.008445 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:12.012506 (+ 4061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:12.012528 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:12.015642 (+ 3114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":266,"threads_started":1}
W20260925 10:33:12.016070 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:12.004257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34289 (local address 127.0.97.62:36691)
0925 10:33:12.004473 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:12.004481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:12.004769 (+ 288us) server_negotiation.cc:407] Connection header received
0925 10:33:12.004929 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:12.004934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:12.004996 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:12.005091 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:12.005797 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:12.006551 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:12.007725 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:12.008128 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:12.008253 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:12.015836 (+ 7583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34289: BlockingRecv error: recv got EOF from 127.0.97.2:34289 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 10:33:12 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:12 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:12.445501 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:12.437457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:34233)
0925 10:33:12.437994 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:12.438009 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:12.438156 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:12.438525 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:12.438529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:12.438947 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:12.439093 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:12.439101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:12.440140 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:12.440147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:12.441147 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:12.441158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:12.441832 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:12.441838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:12.441997 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:12.442617 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:12.442634 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:12.445174 (+ 2540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":308,"threads_started":1}
W20260925 10:33:12.445600 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:12.437684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34233 (local address 127.0.97.62:36691)
0925 10:33:12.437895 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:12.437903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:12.438089 (+ 186us) server_negotiation.cc:407] Connection header received
0925 10:33:12.438273 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:12.438278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:12.438377 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:12.438482 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:12.439250 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:12.440005 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:12.441283 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:12.441689 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:12.441821 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:12.445351 (+ 3530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34233: BlockingRecv error: recv got EOF from 127.0.97.3:34233 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 10:33:12 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:12 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:12.581779 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:12.573679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:60997)
0925 10:33:12.574119 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:12.574135 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:12.574258 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:12.574603 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:12.574608 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:12.575073 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:12.575318 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:12.575327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:12.576462 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:12.576470 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:12.577355 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:12.577365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:12.578015 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:12.578020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:12.578202 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:12.578808 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:12.578825 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:12.581426 (+ 2601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":229,"threads_started":1}
W20260925 10:33:12.581861 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:12.573710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60997 (local address 127.0.97.62:36691)
0925 10:33:12.573931 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:12.573938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:12.574229 (+ 291us) server_negotiation.cc:407] Connection header received
0925 10:33:12.574368 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:12.574371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:12.574445 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:12.574557 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:12.575456 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:12.576332 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:12.577494 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:12.577877 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:12.578007 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:12.581610 (+ 3603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60997: BlockingRecv error: recv got EOF from 127.0.97.1:60997 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 10:33:13 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:13 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:13.025280 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:13.016933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:48993)
0925 10:33:13.017391 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:13.017405 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:13.017557 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:13.017892 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:13.017898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:13.018257 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:13.018456 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:13.018465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:13.019490 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:13.019496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:13.020678 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:13.020696 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:13.021584 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:13.021590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:13.021775 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:13.022334 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:13.022350 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:13.024950 (+ 2600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":239,"threads_started":1}
W20260925 10:33:13.025388 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:13.016988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48993 (local address 127.0.97.62:36691)
0925 10:33:13.017184 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:13.017193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:13.017487 (+ 294us) server_negotiation.cc:407] Connection header received
0925 10:33:13.017695 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:13.017700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:13.017764 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:13.017860 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:13.018597 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:13.019362 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:13.020874 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:13.021443 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:13.021584 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:13.025154 (+ 3570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48993: BlockingRecv error: recv got EOF from 127.0.97.2:48993 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 10:33:13 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:13 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:13.454373 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:13.446412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:38805)
0925 10:33:13.446930 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:13.446946 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:13.447054 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:13.447360 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:13.447363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:13.447686 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:13.447858 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:13.447866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:13.448789 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:13.448793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:13.449692 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:13.449702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:13.450300 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:13.450353 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:13.450485 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:13.451044 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:13.451060 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:13.453992 (+ 2932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":279,"threads_started":1}
W20260925 10:33:13.454432 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:13.446488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38805 (local address 127.0.97.62:36691)
0925 10:33:13.446646 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:13.446650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:13.447016 (+ 366us) server_negotiation.cc:407] Connection header received
0925 10:33:13.447177 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:13.447181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:13.447238 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:13.447316 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:13.447974 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:13.448668 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:13.449806 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:13.450188 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:13.450275 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:13.454192 (+ 3917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38805: BlockingRecv error: recv got EOF from 127.0.97.3:38805 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 10:33:13 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:13 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:13.590567 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:13.582822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:60023)
0925 10:33:13.583365 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:13.583381 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:13.583489 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:13.583761 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:13.583764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:13.584090 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:13.584233 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:13.584240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:13.585141 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:13.585144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:13.586108 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:13.586118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:13.586826 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:13.586830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:13.586949 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:13.587591 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:13.587609 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:13.590292 (+ 2683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":258,"threads_started":1}
W20260925 10:33:13.590752 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:13.582867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60023 (local address 127.0.97.62:36691)
0925 10:33:13.583040 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:13.583044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:13.583447 (+ 403us) server_negotiation.cc:407] Connection header received
0925 10:33:13.583563 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:13.583566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:13.583635 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:13.583711 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:13.584341 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:13.585039 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:13.586223 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:13.586709 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:13.586799 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:13.590508 (+ 3709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60023: BlockingRecv error: recv got EOF from 127.0.97.1:60023 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 10:33:14 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:14 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:14.034286 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:14.026375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:47561)
0925 10:33:14.026892 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:14.026910 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:14.027042 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:14.027440 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:14.027445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:14.027927 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:14.028074 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:14.028085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:14.029022 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:14.029026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:14.030085 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:14.030097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:14.030795 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:14.030799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:14.030933 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:14.031463 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:14.031479 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:14.034071 (+ 2592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":245,"threads_started":1}
W20260925 10:33:14.034489 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:14.026416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47561 (local address 127.0.97.62:36691)
0925 10:33:14.026622 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:14.026629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:14.026994 (+ 365us) server_negotiation.cc:407] Connection header received
0925 10:33:14.027157 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:14.027162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:14.027236 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:14.027369 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:14.028196 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:14.028901 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:14.030203 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:14.030679 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:14.030774 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:14.034228 (+ 3454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47561: BlockingRecv error: recv got EOF from 127.0.97.2:47561 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 10:33:14 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:14 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:14.463693 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:14.455402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:50029)
0925 10:33:14.455913 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:14.455928 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:14.456033 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:14.456318 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:14.456321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:14.456646 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:14.456786 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:14.456793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:14.457829 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:14.457832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:14.458825 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:14.458835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:14.459766 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:14.459772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:14.459988 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:14.460670 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:14.460692 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:14.463475 (+ 2783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":262,"threads_started":1}
W20260925 10:33:14.463939 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:14.455476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50029 (local address 127.0.97.62:36691)
0925 10:33:14.455629 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:14.455634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:14.455998 (+ 364us) server_negotiation.cc:407] Connection header received
0925 10:33:14.456122 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:14.456125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:14.456187 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:14.456266 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:14.456938 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:14.457711 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:14.458983 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:14.459604 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:14.459724 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:14.463653 (+ 3929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50029: BlockingRecv error: recv got EOF from 127.0.97.3:50029 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 10:33:14 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:14 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:14.599534 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:14.591447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:42931)
0925 10:33:14.591906 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:14.591922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:14.592067 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:14.592404 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:14.592409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:14.592805 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:14.592970 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:14.592979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:14.593945 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:14.593951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:14.594917 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:14.594927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:14.595606 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:14.595611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:14.595784 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:14.596416 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:14.596434 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:14.599180 (+ 2746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":225,"threads_started":1}
W20260925 10:33:14.599582 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:14.591512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42931 (local address 127.0.97.62:36691)
0925 10:33:14.591747 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:14.591762 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:14.592002 (+ 240us) server_negotiation.cc:407] Connection header received
0925 10:33:14.592204 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:14.592210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:14.592274 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:14.592371 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:14.593112 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:14.593810 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:14.595053 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:14.595465 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:14.595598 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:14.599367 (+ 3769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42931: BlockingRecv error: recv got EOF from 127.0.97.1:42931 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 10:33:15 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:15 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:15.043253 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:15.035414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:35425)
0925 10:33:15.035926 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:15.035941 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:15.036044 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:15.036293 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:15.036296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:15.036680 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:15.036836 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:15.036844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:15.037762 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:15.037766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:15.038889 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:15.038901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:15.039580 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:15.039586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:15.039721 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:15.040348 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:15.040365 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:15.043027 (+ 2662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":258,"threads_started":1}
W20260925 10:33:15.043433 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:15.035575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35425 (local address 127.0.97.62:36691)
0925 10:33:15.035724 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:15.035728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:15.036007 (+ 279us) server_negotiation.cc:407] Connection header received
0925 10:33:15.036122 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:15.036125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:15.036180 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:15.036256 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:15.036949 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:15.037645 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:15.038999 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:15.039431 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:15.039520 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:15.043190 (+ 3670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35425: BlockingRecv error: recv got EOF from 127.0.97.2:35425 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 10:33:15 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:15 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:15.471915 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:15.464539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:59035)
0925 10:33:15.465001 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:15.465018 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:15.465123 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:15.465401 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:15.465404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:15.465758 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:15.465921 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:15.465930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:15.466903 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:15.466907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:15.467790 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:15.467799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:15.468415 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:15.468419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:15.468528 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:15.469062 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:15.469078 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:15.471571 (+ 2493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":225,"threads_started":1}
W20260925 10:33:15.471971 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:15.464579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59035 (local address 127.0.97.62:36691)
0925 10:33:15.464791 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:15.464797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:15.465095 (+ 298us) server_negotiation.cc:407] Connection header received
0925 10:33:15.465216 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:15.465219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:15.465275 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:15.465351 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:15.466038 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:15.466763 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:15.467901 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:15.468305 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:15.468394 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:15.471784 (+ 3390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59035: BlockingRecv error: recv got EOF from 127.0.97.3:59035 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 10:33:15 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:15 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:15.608904 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:15.600701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:56587)
0925 10:33:15.601249 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:15.601264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:15.601427 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:15.601804 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:15.601809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:15.602312 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:15.602575 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:15.602586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:15.603603 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:15.603607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:15.604476 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:15.604486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:15.605249 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:15.605253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:15.605369 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:15.605981 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:15.606001 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:15.608690 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":257,"threads_started":1}
W20260925 10:33:15.609176 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:15.600751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56587 (local address 127.0.97.62:36691)
0925 10:33:15.601046 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:15.601052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:15.601369 (+ 317us) server_negotiation.cc:407] Connection header received
0925 10:33:15.601566 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:15.601576 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:15.601650 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:15.601781 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:15.602738 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:15.603485 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:15.604594 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:15.605123 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:15.605215 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:15.608872 (+ 3657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56587: BlockingRecv error: recv got EOF from 127.0.97.1:56587 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Sep 25 10:33:16 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:16 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:16.052560 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:16.044166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:36721)
0925 10:33:16.044673 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:16.044689 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:16.044837 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:16.045182 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:16.045187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:16.045582 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:16.045727 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:16.045735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:16.046813 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:16.046819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:16.047709 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:16.047719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:16.048414 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:16.048420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:16.048591 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:16.049307 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:16.049329 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:16.052340 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":286,"threads_started":1}
W20260925 10:33:16.052789 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:16.044253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36721 (local address 127.0.97.62:36691)
0925 10:33:16.044499 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:16.044507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:16.044774 (+ 267us) server_negotiation.cc:407] Connection header received
0925 10:33:16.044954 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:16.044962 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:16.045035 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:16.045149 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:16.045883 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:16.046682 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:16.047850 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:16.048261 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:16.048402 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:16.052554 (+ 4152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36721: BlockingRecv error: recv got EOF from 127.0.97.2:36721 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 10:33:16 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:16 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:16.480737 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:16.472941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:51571)
0925 10:33:16.473535 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:16.473556 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:16.473692 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:16.474055 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:16.474065 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:16.474624 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:16.474768 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:16.474776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:16.475760 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:16.475764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:16.476644 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:16.476654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:16.477303 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:16.477307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:16.477420 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:16.477961 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:16.477977 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:16.480529 (+ 2552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":329,"threads_started":1}
W20260925 10:33:16.480918 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:16.473002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51571 (local address 127.0.97.62:36691)
0925 10:33:16.473185 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:16.473191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:16.473645 (+ 454us) server_negotiation.cc:407] Connection header received
0925 10:33:16.473804 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:16.473808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:16.473901 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:16.474020 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:16.474889 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:16.475646 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:16.476776 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:16.477174 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:16.477272 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:16.480707 (+ 3435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51571: BlockingRecv error: recv got EOF from 127.0.97.3:51571 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 10:33:16 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:16 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:16.617998 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:16.609776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:55193)
0925 10:33:16.610428 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:16.610444 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:16.610551 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:16.610824 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:16.610827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:16.611228 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:16.611388 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:16.611396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:16.612310 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:16.612313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:16.613188 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:16.613197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:16.614101 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:16.614106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:16.614285 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:16.615048 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:16.615069 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:16.617770 (+ 2701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":304,"threads_started":1}
W20260925 10:33:16.618223 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:16.609836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55193 (local address 127.0.97.62:36691)
0925 10:33:16.610044 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:16.610049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:16.610520 (+ 471us) server_negotiation.cc:407] Connection header received
0925 10:33:16.610645 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:16.610648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:16.610703 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:16.610781 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:16.611510 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:16.612191 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:16.613318 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:16.613939 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:16.614040 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:16.617990 (+ 3950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55193: BlockingRecv error: recv got EOF from 127.0.97.1:55193 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 10:33:17 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:17 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:17.068289 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:17.053568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:51123)
0925 10:33:17.054126 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:17.054143 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:17.054255 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:17.054614 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:17.054617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:17.054970 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:17.055169 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:17.055177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:17.056116 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:17.056120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:17.056998 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:17.057008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:17.057653 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:17.057657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:17.057761 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:17.065120 (+ 7359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:17.065139 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:17.067936 (+ 2797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":245,"threads_started":1}
W20260925 10:33:17.068393 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:17.053663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51123 (local address 127.0.97.62:36691)
0925 10:33:17.053879 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:17.053888 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:17.054219 (+ 331us) server_negotiation.cc:407] Connection header received
0925 10:33:17.054409 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:17.054412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:17.054470 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:17.054549 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:17.055284 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:17.055978 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:17.057106 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:17.057535 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:17.057620 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:17.068135 (+ 10515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51123: BlockingRecv error: recv got EOF from 127.0.97.2:51123 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 10:33:17 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:17 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:17.489471 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:17.481718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:35899)
0925 10:33:17.482231 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:17.482246 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:17.482383 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:17.482661 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:17.482664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:17.483026 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:17.483190 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:17.483198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:17.484143 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:17.484146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:17.485044 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:17.485054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:17.485717 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:17.485721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:17.485837 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:17.486395 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:17.486411 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:17.489239 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":188,"threads_started":1}
W20260925 10:33:17.489600 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:17.481740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35899 (local address 127.0.97.62:36691)
0925 10:33:17.481900 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:17.481904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:17.482326 (+ 422us) server_negotiation.cc:407] Connection header received
0925 10:33:17.482469 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:17.482473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:17.482533 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:17.482625 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:17.483318 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:17.484029 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:17.485172 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:17.485596 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:17.485687 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:17.489400 (+ 3713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35899: BlockingRecv error: recv got EOF from 127.0.97.3:35899 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 10:33:17 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:17 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:17.626516 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:17.619036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:52401)
0925 10:33:17.619517 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:17.619536 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:17.619670 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:17.620062 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:17.620065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:17.620439 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:17.620579 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:17.620586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:17.621530 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:17.621533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:17.622459 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:17.622469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:17.623149 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:17.623153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:17.623282 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:17.623824 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:17.623840 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:17.626283 (+ 2443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":366,"threads_started":1}
W20260925 10:33:17.626717 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:17.619209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52401 (local address 127.0.97.62:36691)
0925 10:33:17.619397 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:17.619405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:17.619613 (+ 208us) server_negotiation.cc:407] Connection header received
0925 10:33:17.619824 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:17.619829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:17.619909 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:17.620002 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:17.620700 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:17.621400 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:17.622576 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:17.623005 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:17.623099 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:17.626487 (+ 3388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52401: BlockingRecv error: recv got EOF from 127.0.97.1:52401 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 10:33:18 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:18 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:18.077349 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:18.069137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:43023)
0925 10:33:18.069608 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:18.069623 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:18.069732 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:18.070042 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:18.070045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:18.070438 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:18.070580 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:18.070588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:18.071695 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:18.071698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:18.072701 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:18.072712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:18.073526 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:18.073530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:18.073638 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:18.074189 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:18.074204 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:18.077118 (+ 2914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":266,"threads_started":1}
W20260925 10:33:18.077509 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:18.069316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43023 (local address 127.0.97.62:36691)
0925 10:33:18.069541 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:18.069547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:18.069713 (+ 166us) server_negotiation.cc:407] Connection header received
0925 10:33:18.069850 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:18.069853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:18.069925 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:18.070000 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:18.070710 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:18.071560 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:18.072849 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:18.073372 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:18.073480 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:18.077297 (+ 3817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43023: BlockingRecv error: recv got EOF from 127.0.97.2:43023 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 25 10:33:18 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:18 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:18.498699 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:18.490391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:43865)
0925 10:33:18.490891 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:18.490907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:18.491052 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:18.491409 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:18.491413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:18.491835 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:18.491986 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:18.491993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:18.492976 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:18.492983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:18.493939 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:18.493949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:18.494734 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:18.494740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:18.494917 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:18.495533 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:18.495549 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:18.498314 (+ 2765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":305,"threads_started":1}
W20260925 10:33:18.498809 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:18.490456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43865 (local address 127.0.97.62:36691)
0925 10:33:18.490674 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:18.490681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:18.490993 (+ 312us) server_negotiation.cc:407] Connection header received
0925 10:33:18.491206 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:18.491211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:18.491279 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:18.491386 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:18.492124 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:18.492838 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:18.494076 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:18.494563 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:18.494701 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:18.498526 (+ 3825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43865: BlockingRecv error: recv got EOF from 127.0.97.3:43865 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 10:33:18 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:18 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:18.635260 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:18.627427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:48037)
0925 10:33:18.627954 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:18.627970 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:18.628076 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:18.628374 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:18.628377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:18.628747 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:18.628908 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:18.628916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:18.629862 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:18.629866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:18.630930 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:18.630942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:18.631588 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:18.631592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:18.631706 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:18.632270 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:18.632286 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:18.635011 (+ 2725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":296,"threads_started":1}
W20260925 10:33:18.635416 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:18.627476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48037 (local address 127.0.97.62:36691)
0925 10:33:18.627705 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:18.627713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:18.628049 (+ 336us) server_negotiation.cc:407] Connection header received
0925 10:33:18.628155 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:18.628159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:18.628229 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:18.628322 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:18.629024 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:18.629745 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:18.631040 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:18.631444 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:18.631575 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:18.635216 (+ 3641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48037: BlockingRecv error: recv got EOF from 127.0.97.1:48037 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 10:33:19 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:19 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:19.086015 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:19.078138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:44323)
0925 10:33:19.078634 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:19.078650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:19.078773 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:19.079082 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:19.079086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:19.079449 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:19.079618 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:19.079625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:19.080596 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:19.080600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:19.081525 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:19.081536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:19.082201 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:19.082206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:19.082403 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:19.082939 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:19.082956 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:19.085661 (+ 2705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":296,"threads_started":1}
W20260925 10:33:19.086199 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:19.078202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44323 (local address 127.0.97.62:36691)
0925 10:33:19.078458 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:19.078464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:19.078745 (+ 281us) server_negotiation.cc:407] Connection header received
0925 10:33:19.078857 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:19.078860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:19.078918 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:19.079004 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:19.079742 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:19.080465 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:19.081644 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:19.082066 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:19.082152 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:19.085858 (+ 3706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44323: BlockingRecv error: recv got EOF from 127.0.97.2:44323 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 10:33:19 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:19 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:19.508050 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:19.499561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:55711)
0925 10:33:19.500049 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:19.500064 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:19.500231 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:19.500566 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:19.500570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:19.500950 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:19.501113 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:19.501122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:19.502124 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:19.502131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:19.503171 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:19.503182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:19.503888 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:19.503894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:19.504064 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:19.504587 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:19.504604 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:19.507704 (+ 3100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":293,"threads_started":1}
W20260925 10:33:19.508102 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:19.499645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55711 (local address 127.0.97.62:36691)
0925 10:33:19.499850 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:19.499858 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:19.500143 (+ 285us) server_negotiation.cc:407] Connection header received
0925 10:33:19.500368 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:19.500379 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:19.500435 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:19.500540 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:19.501243 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:19.501999 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:19.503309 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:19.503743 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:19.503877 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:19.507886 (+ 4009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55711: BlockingRecv error: recv got EOF from 127.0.97.3:55711 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 10:33:19 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:19 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:19.644409 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:19.636227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:48703)
0925 10:33:19.636722 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:19.636740 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:19.636891 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:19.637311 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:19.637317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:19.637764 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:19.637941 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:19.637956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:19.639004 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:19.639011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:19.640102 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:19.640112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:19.640778 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:19.640787 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:19.640917 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:19.641535 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:19.641551 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:19.644179 (+ 2628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":404,"threads_started":1}
W20260925 10:33:19.644606 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:19.636440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48703 (local address 127.0.97.62:36691)
0925 10:33:19.636624 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:19.636631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:19.636836 (+ 205us) server_negotiation.cc:407] Connection header received
0925 10:33:19.637021 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:19.637029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:19.637140 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:19.637251 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:19.638084 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:19.638855 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:19.640213 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:19.640626 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:19.640763 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:19.644345 (+ 3582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48703: BlockingRecv error: recv got EOF from 127.0.97.1:48703 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 10:33:20 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:20 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:20.094967 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:20.086952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:33391)
0925 10:33:20.087440 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:20.087456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:20.087623 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:20.087939 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:20.087943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:20.088418 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:20.088564 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:20.088573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:20.089590 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:20.089596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:20.090572 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:20.090583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:20.091324 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:20.091330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:20.091508 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:20.092051 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:20.092067 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:20.094745 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":280,"threads_started":1}
W20260925 10:33:20.095134 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:20.087041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33391 (local address 127.0.97.62:36691)
0925 10:33:20.087204 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:20.087209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:20.087563 (+ 354us) server_negotiation.cc:407] Connection header received
0925 10:33:20.087722 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:20.087725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:20.087780 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:20.087864 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:20.088721 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:20.089451 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:20.090716 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:20.091169 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:20.091312 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:20.094922 (+ 3610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33391: BlockingRecv error: recv got EOF from 127.0.97.2:33391 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 10:33:20 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:20 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:20.516912 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:20.508992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:36305)
0925 10:33:20.509508 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:20.509524 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:20.509666 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:20.510005 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:20.510010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:20.510536 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:20.510719 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:20.510728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:20.511727 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:20.511731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:20.512607 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:20.512623 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:20.513258 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:20.513265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:20.513409 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:20.514036 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:20.514052 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:20.516713 (+ 2661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":280,"threads_started":1}
W20260925 10:33:20.517128 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:20.509226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36305 (local address 127.0.97.62:36691)
0925 10:33:20.509426 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:20.509433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:20.509605 (+ 172us) server_negotiation.cc:407] Connection header received
0925 10:33:20.509791 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:20.509801 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:20.509858 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:20.509933 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:20.510838 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:20.511611 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:20.512725 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:20.513119 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:20.513204 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:20.516882 (+ 3678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36305: BlockingRecv error: recv got EOF from 127.0.97.3:36305 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 10:33:20 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:20 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:20.653816 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:20.645290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:54433)
0925 10:33:20.645821 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:20.645837 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:20.645945 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:20.646264 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:20.646267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:20.646678 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:20.646850 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:20.646857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:20.648248 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:20.648253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:20.649331 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:20.649342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:20.650156 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:20.650160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:20.650276 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:20.650890 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:20.650907 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:20.653585 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":287,"threads_started":1}
W20260925 10:33:20.653945 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:20.645333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54433 (local address 127.0.97.62:36691)
0925 10:33:20.645539 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:20.645543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:20.645909 (+ 366us) server_negotiation.cc:407] Connection header received
0925 10:33:20.646036 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:20.646040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:20.646138 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:20.646220 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:20.646984 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:20.648091 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:20.649460 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:20.650012 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:20.650108 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:20.653757 (+ 3649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54433: BlockingRecv error: recv got EOF from 127.0.97.1:54433 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 25 10:33:21 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:21 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:21.103603 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:21.095838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:54049)
0925 10:33:21.096302 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:21.096318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:21.096463 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:21.096794 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:21.096799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:21.097186 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:21.097351 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:21.097360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:21.098356 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:21.098360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:21.099179 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:21.099189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:21.099844 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:21.099849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:21.100017 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:21.100560 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:21.100575 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:21.103374 (+ 2799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":256,"threads_started":1}
W20260925 10:33:21.103751 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:21.095905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54049 (local address 127.0.97.62:36691)
0925 10:33:21.096114 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:21.096122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:21.096396 (+ 274us) server_negotiation.cc:407] Connection header received
0925 10:33:21.096579 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:21.096584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:21.096650 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:21.096762 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:21.097482 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:21.098187 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:21.099320 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:21.099699 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:21.099832 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:21.103546 (+ 3714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54049: BlockingRecv error: recv got EOF from 127.0.97.2:54049 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 10:33:21 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:21 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:21.525667 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:21.517669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:42853)
0925 10:33:21.518184 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:21.518207 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:21.518373 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:21.518755 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:21.518762 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:21.519307 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:21.519483 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:21.519492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:21.520431 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:21.520434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:21.521338 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:21.521348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:21.521965 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:21.521969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:21.522085 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:21.522663 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:21.522679 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:21.525385 (+ 2706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":255,"threads_started":1}
W20260925 10:33:21.525889 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:21.517829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42853 (local address 127.0.97.62:36691)
0925 10:33:21.518054 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:21.518060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:21.518281 (+ 221us) server_negotiation.cc:407] Connection header received
0925 10:33:21.518492 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:21.518497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:21.518579 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:21.518680 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:21.519606 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:21.520310 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:21.521452 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:21.521830 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:21.521917 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:21.525640 (+ 3723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42853: BlockingRecv error: recv got EOF from 127.0.97.3:42853 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 10:33:21 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:21 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:21.662912 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:21.654841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:46565)
0925 10:33:21.655372 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:21.655388 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:21.655529 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:21.655914 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:21.655918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:21.656320 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:21.656459 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:21.656468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:21.657473 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:21.657478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:21.658491 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:21.658502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:21.659210 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:21.659216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:21.659388 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:21.659943 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:21.659960 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:21.662694 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":320,"threads_started":1}
W20260925 10:33:21.663058 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:21.654896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46565 (local address 127.0.97.62:36691)
0925 10:33:21.655096 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:21.655101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:21.655447 (+ 346us) server_negotiation.cc:407] Connection header received
0925 10:33:21.655694 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:21.655699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:21.655780 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:21.655880 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:21.656607 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:21.657337 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:21.658619 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:21.659071 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:21.659209 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:21.662848 (+ 3639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46565: BlockingRecv error: recv got EOF from 127.0.97.1:46565 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 10:33:22 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:22 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:22.117683 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:22.104555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:51795)
0925 10:33:22.105122 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:22.105139 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:22.105246 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:22.105521 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:22.105524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:22.105984 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:22.106159 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:22.106170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:22.107147 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:22.107151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:22.108045 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:22.108055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:22.108738 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:22.108774 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:22.108918 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:22.114535 (+ 5617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:22.114552 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:22.117357 (+ 2805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":358,"threads_started":1}
W20260925 10:33:22.117748 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:22.104638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51795 (local address 127.0.97.62:36691)
0925 10:33:22.104805 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:22.104809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:22.105205 (+ 396us) server_negotiation.cc:407] Connection header received
0925 10:33:22.105346 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:22.105349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:22.105406 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:22.105485 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:22.106289 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:22.107020 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:22.108162 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:22.108593 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:22.108696 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:22.117546 (+ 8850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51795: BlockingRecv error: recv got EOF from 127.0.97.2:51795 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 10:33:22 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:22 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:22.534636 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:22.526633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:53647)
0925 10:33:22.527078 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:22.527094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:22.527208 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:22.527540 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:22.527544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:22.527921 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:22.528067 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:22.528075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:22.529001 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:22.529005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:22.529923 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:22.529932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:22.530909 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:22.530914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:22.531036 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:22.531566 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:22.531582 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:22.534408 (+ 2826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":243,"threads_started":1}
W20260925 10:33:22.534802 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:22.526683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53647 (local address 127.0.97.62:36691)
0925 10:33:22.526928 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:22.526935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:22.527179 (+ 244us) server_negotiation.cc:407] Connection header received
0925 10:33:22.527320 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:22.527324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:22.527388 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:22.527468 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:22.528187 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:22.528881 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:22.530054 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:22.530777 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:22.530883 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:22.534591 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53647: BlockingRecv error: recv got EOF from 127.0.97.3:53647 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 10:33:22 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:22 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:22.671958 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:22.663917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:43461)
0925 10:33:22.664402 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:22.664416 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:22.664563 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:22.664918 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:22.664923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:22.665310 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:22.665470 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:22.665477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:22.666509 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:22.666512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:22.667428 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:22.667438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:22.668185 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:22.668191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:22.668346 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:22.668979 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:22.668997 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:22.671596 (+ 2599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":252,"threads_started":1}
W20260925 10:33:22.672008 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:22.663996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43461 (local address 127.0.97.62:36691)
0925 10:33:22.664213 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:22.664221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:22.664500 (+ 279us) server_negotiation.cc:407] Connection header received
0925 10:33:22.664685 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:22.664690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:22.664774 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:22.664875 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:22.665608 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:22.666350 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:22.667566 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:22.667985 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:22.668121 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:22.671778 (+ 3657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43461: BlockingRecv error: recv got EOF from 127.0.97.1:43461 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 10:33:23 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:23 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:23.126297 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:23.118626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:38115)
0925 10:33:23.119152 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:23.119166 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:23.119271 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:23.119562 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:23.119566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:23.119936 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:23.120100 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:23.120108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:23.121072 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:23.121075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:23.121972 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:23.121982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:23.122834 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:23.122839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:23.122950 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:23.123478 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:23.123494 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:23.126064 (+ 2570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":266,"threads_started":1}
W20260925 10:33:23.126556 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:23.118735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38115 (local address 127.0.97.62:36691)
0925 10:33:23.118900 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:23.118904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:23.119245 (+ 341us) server_negotiation.cc:407] Connection header received
0925 10:33:23.119364 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:23.119368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:23.119443 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:23.119525 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:23.120218 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:23.120960 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:23.122104 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:23.122678 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:23.122770 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:23.126242 (+ 3472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38115: BlockingRecv error: recv got EOF from 127.0.97.2:38115 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 10:33:23 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:23 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:23.543399 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:23.535544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:43669)
0925 10:33:23.536066 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:23.536083 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:23.536186 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:23.536444 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:23.536448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:23.536814 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:23.536980 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:23.536988 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:23.537923 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:23.537927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:23.538978 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:23.538990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:23.539670 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:23.539674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:23.539864 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:23.540428 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:23.540451 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:23.543182 (+ 2731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":269,"threads_started":1}
W20260925 10:33:23.543545 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:23.535607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43669 (local address 127.0.97.62:36691)
0925 10:33:23.535786 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:23.535793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:23.536139 (+ 346us) server_negotiation.cc:407] Connection header received
0925 10:33:23.536262 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:23.536265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:23.536322 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:23.536396 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:23.537094 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:23.537808 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:23.539109 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:23.539523 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:23.539612 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:23.543365 (+ 3753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43669: BlockingRecv error: recv got EOF from 127.0.97.3:43669 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 10:33:23 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:23 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:23.680841 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:23.672885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:39915)
0925 10:33:23.673366 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:23.673387 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:23.673506 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:23.673793 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:23.673795 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:23.674176 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:23.674394 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:23.674404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:23.675789 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:23.675794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:23.676727 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:23.676737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:23.677350 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:23.677354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:23.677475 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:23.678051 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:23.678067 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:23.680602 (+ 2535us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":300,"threads_started":1}
W20260925 10:33:23.681007 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:23.672995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39915 (local address 127.0.97.62:36691)
0925 10:33:23.673271 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:23.673275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:23.673465 (+ 190us) server_negotiation.cc:407] Connection header received
0925 10:33:23.673596 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:23.673600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:23.673669 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:23.673748 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:23.674551 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:23.675643 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:23.676849 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:23.677230 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:23.677321 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:23.680791 (+ 3470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39915: BlockingRecv error: recv got EOF from 127.0.97.1:39915 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
Sep 25 10:33:24 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:24 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:24.135144 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:24.127299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:39207)
0925 10:33:24.127741 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:24.127757 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:24.127865 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:24.128153 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:24.128156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:24.128497 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:24.128648 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:24.128655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:24.129588 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:24.129592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:24.130550 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:24.130561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:24.131214 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:24.131219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:24.131378 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:24.132124 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:24.132140 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:24.134908 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":251,"threads_started":1}
W20260925 10:33:24.135273 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:24.127400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39207 (local address 127.0.97.62:36691)
0925 10:33:24.127642 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:24.127648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:24.127840 (+ 192us) server_negotiation.cc:407] Connection header received
0925 10:33:24.127958 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:24.127962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:24.128033 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:24.128112 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:24.128760 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:24.129475 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:24.130666 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:24.131080 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:24.131168 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:24.135082 (+ 3914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39207: BlockingRecv error: recv got EOF from 127.0.97.2:39207 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 25 10:33:24 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:24 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:24.551930 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:24.544275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.3:40437)
0925 10:33:24.544710 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:24.544728 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:24.544843 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:24.545111 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:24.545114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:24.545451 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:24.545591 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:24.545598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:24.546598 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:24.546602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:24.547497 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:24.547507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:24.548182 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:24.548186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:24.548299 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:24.548843 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:24.548860 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:24.551566 (+ 2706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":231,"threads_started":1}
W20260925 10:33:24.552017 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:24.544339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40437 (local address 127.0.97.62:36691)
0925 10:33:24.544477 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:24.544482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:24.544801 (+ 319us) server_negotiation.cc:407] Connection header received
0925 10:33:24.544940 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:24.544944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:24.544997 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:24.545071 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:24.545706 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:24.546476 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:24.547630 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:24.548061 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:24.548151 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:24.551798 (+ 3647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40437: BlockingRecv error: recv got EOF from 127.0.97.3:40437 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 10:33:24 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:24 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:24.689918 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:24.682146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.1:46089)
0925 10:33:24.682656 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:24.682671 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:24.682773 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:24.683080 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:24.683083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:24.683472 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:24.683644 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:24.683652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:24.684658 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:24.684662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:24.685570 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:24.685580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:24.686370 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:24.686382 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:24.686495 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:24.687090 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:24.687107 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:24.689648 (+ 2541us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":316,"threads_started":1}
W20260925 10:33:24.690011 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:24.682170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46089 (local address 127.0.97.62:36691)
0925 10:33:24.682426 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:24.682434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:24.682753 (+ 319us) server_negotiation.cc:407] Connection header received
0925 10:33:24.682865 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:24.682869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:24.682937 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:24.683034 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:24.683787 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:24.684538 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:24.685701 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:24.686142 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:24.686265 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:24.689818 (+ 3553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46089: BlockingRecv error: recv got EOF from 127.0.97.1:46089 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 10:33:25 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:33:25 dist-test-slave-dxdd krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:33:25.144909 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:25.136219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36691 (local address 127.0.97.2:44211)
0925 10:33:25.136686 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:25.136702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:25.136811 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:25.137144 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:25.137148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:25.137704 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:25.137922 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:25.137937 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:25.139209 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:25.139213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:25.140187 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:25.140198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:25.141021 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:25.141027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:25.141190 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:25.141873 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:25.141889 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:25.144538 (+ 2649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36691: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":257,"threads_started":1}
W20260925 10:33:25.144946 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:25.136294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44211 (local address 127.0.97.62:36691)
0925 10:33:25.136513 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:25.136520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:25.136787 (+ 267us) server_negotiation.cc:407] Connection header received
0925 10:33:25.136942 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:25.136946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:25.137003 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:25.137119 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:25.138085 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:25.139075 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:25.140308 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:25.140858 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:25.141015 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:25.144715 (+ 3700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44211: BlockingRecv error: recv got EOF from 127.0.97.2:44211 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:33:25.407609 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 9219
I20260925 10:33:25.419529 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:25.408554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33038 (local address 127.0.97.1:34709)
0925 10:33:25.409076 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:25.409083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:25.409108 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:33:25.409163 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:25.409169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:25.409399 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:25.409545 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:25.411247 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:25.412246 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:25.413440 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:25.414002 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:25.414141 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:25.416547 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:25.416555 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:25.416569 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:25.416642 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:25.418911 (+ 2269us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 10:33:25.419218 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33038: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":263,"threads_started":1}
W20260925 10:33:25.419592 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 9219: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:34709: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 10:33:25.419641 9679 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33038: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 10:33:25.419791 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 9219
I20260925 10:33:25.436087 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 9370
I20260925 10:33:25.447533 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:25.437048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49724 (local address 127.0.97.2:38523)
0925 10:33:25.437666 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:25.437673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:25.437703 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:33:25.437760 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:25.437766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:25.437987 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:25.438141 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:25.439437 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:25.440485 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:25.441790 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:25.442371 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:25.442470 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:25.444623 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:25.444630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:25.444643 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:25.444674 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:25.447072 (+ 2398us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 10:33:25.447303 (+ 231us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49724: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":327,"threads_started":1}
W20260925 10:33:25.447645 9680 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49724: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 10:33:25.447763 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 9370: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:38523: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 10:33:25.447919 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 9370
I20260925 10:33:25.464221 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 9501
I20260925 10:33:25.475809 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:25.465110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50368 (local address 127.0.97.3:33745)
0925 10:33:25.465686 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:25.465699 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:25.465727 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:33:25.465797 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:25.465804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:25.466019 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:25.466178 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:25.467395 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:25.468361 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:25.469432 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:25.469959 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:25.470049 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:25.472748 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:25.472756 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:25.472771 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:25.472801 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:25.475339 (+ 2538us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 10:33:25.475574 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50368: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":319,"threads_started":1}
W20260925 10:33:25.475924 9681 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50368: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 10:33:25.475925 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 9501: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:33745: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 10:33:25.476066 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 9501
I20260925 10:33:25.491897 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 9150
I20260925 10:33:25.493036 9212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:25.606415 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 9150
2026-09-25T10:33:25Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20960 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T10:33:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:33:25Z Disabled control of system clock
I20260925 10:33:25.680184 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35091
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 10:33:25.997153 9686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:25.997435 9686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:25.997515 9686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:26.004601 9686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:33:26.004704 9686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:26.004742 9686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:26.004791 9686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:33:26.004827 9686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:33:26.013736 9686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:26.014972 9686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:26.016644 9686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:26.028363 9691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:26.028312 9692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:26.028384 9694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:26.029141 9686 server_base.cc:1034] running on GCE node
I20260925 10:33:26.029954 9686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:26.031486 9686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:26.032709 9686 hybrid_clock.cc:648] HybridClock initialized: now 1790332406032664 us; error 45 us; skew 500 ppm
I20260925 10:33:26.033053 9686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:33:26.035456 9686 webserver.cc:460] Webserver started at http://127.0.97.62:38587/ using document root <none> and password file <none>
I20260925 10:33:26.036140 9686 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:26.036286 9686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:26.036674 9686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:26.039587 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "e930b9486c054f66922b15afa3be8863"
format_stamp: "Formatted at 2026-09-25 10:33:26 on dist-test-slave-dxdd"
server_key: "5012a0aa93536eac1538ede1c2ebe3c6"
server_key_iv: "6089c150daa159074fb309ee5ca0df4b"
server_key_version: "encryptionkey@0"
I20260925 10:33:26.040369 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "e930b9486c054f66922b15afa3be8863"
format_stamp: "Formatted at 2026-09-25 10:33:26 on dist-test-slave-dxdd"
server_key: "5012a0aa93536eac1538ede1c2ebe3c6"
server_key_iv: "6089c150daa159074fb309ee5ca0df4b"
server_key_version: "encryptionkey@0"
I20260925 10:33:26.045444 9686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:33:26.048873 9700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:26.050122 9686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:33:26.050344 9686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "e930b9486c054f66922b15afa3be8863"
format_stamp: "Formatted at 2026-09-25 10:33:26 on dist-test-slave-dxdd"
server_key: "5012a0aa93536eac1538ede1c2ebe3c6"
server_key_iv: "6089c150daa159074fb309ee5ca0df4b"
server_key_version: "encryptionkey@0"
I20260925 10:33:26.050578 9686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:26.071731 9686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:26.072553 9686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:26.072847 9686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:26.090261 9686 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35091
I20260925 10:33:26.090268 9751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35091 every 8 connection(s)
I20260925 10:33:26.091559 9686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:33:26.095091 9752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:26.097841 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 9686
I20260925 10:33:26.098033 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:33:26.098757 388 external_mini_cluster.cc:1428] Setting key 7a388a80b97944863f12c7cbe8c1c9ec
I20260925 10:33:26.107986 9752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863: Bootstrap starting.
I20260925 10:33:26.111555 9752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:26.112670 9752 log.cc:824] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863: Log is configured to *not* fsync() on all Append() calls
I20260925 10:33:26.113292 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:26.100730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.0.97.62:35091)
0925 10:33:26.101742 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:26.101759 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:26.101807 (+ 48us) server_negotiation.cc:407] Connection header received
0925 10:33:26.103976 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:26.104007 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:26.104559 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:26.104894 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:26.105550 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:26.106862 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:26.108629 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:26.109298 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:26.109452 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:26.110807 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:26.110819 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:26.110832 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:33:26.111356 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:26.111455 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:33:26.111477 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:26.111587 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:26.111856 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 10:33:26.112190 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":299,"threads_started":1}
I20260925 10:33:26.115564 9752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863: No bootstrap required, opened a new log
I20260925 10:33:26.122437 9752 raft_consensus.cc:348] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e930b9486c054f66922b15afa3be8863" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35091 } }
I20260925 10:33:26.122819 9752 raft_consensus.cc:374] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:26.122933 9752 raft_consensus.cc:729] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e930b9486c054f66922b15afa3be8863, State: Initialized, Role: FOLLOWER
I20260925 10:33:26.123437 9752 consensus_queue.cc:260] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e930b9486c054f66922b15afa3be8863" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35091 } }
I20260925 10:33:26.123680 9752 raft_consensus.cc:388] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:33:26.123819 9752 raft_consensus.cc:482] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:33:26.123993 9752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:26.126143 9752 raft_consensus.cc:504] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e930b9486c054f66922b15afa3be8863" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35091 } }
I20260925 10:33:26.126645 9752 leader_election.cc:302] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e930b9486c054f66922b15afa3be8863; no voters:
I20260925 10:33:26.127434 9752 leader_election.cc:288] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:33:26.127545 9757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:33:26.128334 9757 raft_consensus.cc:686] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [term 1 LEADER]: Becoming Leader. State: Replica: e930b9486c054f66922b15afa3be8863, State: Running, Role: LEADER
I20260925 10:33:26.128912 9757 consensus_queue.cc:237] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e930b9486c054f66922b15afa3be8863" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35091 } }
I20260925 10:33:26.129693 9752 sys_catalog.cc:564] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:33:26.131732 9758 sys_catalog.cc:455] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e930b9486c054f66922b15afa3be8863. Latest consensus state: current_term: 1 leader_uuid: "e930b9486c054f66922b15afa3be8863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e930b9486c054f66922b15afa3be8863" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35091 } } }
I20260925 10:33:26.132067 9758 sys_catalog.cc:458] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [sys.catalog]: This master's current role is: LEADER
I20260925 10:33:26.131840 9759 sys_catalog.cc:455] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e930b9486c054f66922b15afa3be8863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e930b9486c054f66922b15afa3be8863" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35091 } } }
I20260925 10:33:26.133102 9759 sys_catalog.cc:458] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863 [sys.catalog]: This master's current role is: LEADER
I20260925 10:33:26.135152 9764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:33:26.140069 9764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:33:26.148043 9764 catalog_manager.cc:1347] Generated new cluster ID: dfaaf314acef4db78ecc154534ba0eba
I20260925 10:33:26.148168 9764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:33:26.159363 9764 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:33:26.160614 9764 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:33:26.169878 9764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e930b9486c054f66922b15afa3be8863: Generated new TSK 0
I20260925 10:33:26.170745 9764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:33:26.186269 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35091
--builtin_ntp_servers=127.0.97.20:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 10:33:26.510588 9776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:26.510841 9776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:26.510982 9776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:26.517951 9776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:26.518042 9776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:26.518141 9776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:33:26.526962 9776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:26.528196 9776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:26.530073 9776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:26.543134 9781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:26.543097 9784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:26.543057 9782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:26.543784 9776 server_base.cc:1034] running on GCE node
I20260925 10:33:26.544457 9776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:26.545327 9776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:26.546552 9776 hybrid_clock.cc:648] HybridClock initialized: now 1790332406546505 us; error 52 us; skew 500 ppm
I20260925 10:33:26.546833 9776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:33:26.549046 9776 webserver.cc:460] Webserver started at http://127.0.97.1:37355/ using document root <none> and password file <none>
I20260925 10:33:26.549636 9776 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:26.549721 9776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:26.549986 9776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:26.552922 9776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "3b45968edbee4e36b85b931a30e9732d"
format_stamp: "Formatted at 2026-09-25 10:33:26 on dist-test-slave-dxdd"
server_key: "ec9400eb3f528f8047d89011ad0ef117"
server_key_iv: "54771ec35e491cb5f5e31e95eef7e75d"
server_key_version: "encryptionkey@0"
I20260925 10:33:26.553678 9776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3b45968edbee4e36b85b931a30e9732d"
format_stamp: "Formatted at 2026-09-25 10:33:26 on dist-test-slave-dxdd"
server_key: "ec9400eb3f528f8047d89011ad0ef117"
server_key_iv: "54771ec35e491cb5f5e31e95eef7e75d"
server_key_version: "encryptionkey@0"
I20260925 10:33:26.558630 9776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:33:26.562013 9790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:26.563225 9776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:33:26.563431 9776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "3b45968edbee4e36b85b931a30e9732d"
format_stamp: "Formatted at 2026-09-25 10:33:26 on dist-test-slave-dxdd"
server_key: "ec9400eb3f528f8047d89011ad0ef117"
server_key_iv: "54771ec35e491cb5f5e31e95eef7e75d"
server_key_version: "encryptionkey@0"
I20260925 10:33:26.563632 9776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:26.582813 9776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:26.583560 9776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:26.583824 9776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:26.584671 9776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:33:26.586030 9776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:26.586153 9776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:26.586284 9776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:26.586395 9776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 10:33:26.618830 9776 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32917
I20260925 10:33:26.618885 9902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32917 every 8 connection(s)
I20260925 10:33:26.620105 9776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:33:26.630198 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 9776
I20260925 10:33:26.630402 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:33:26.631069 388 external_mini_cluster.cc:1428] Setting key c6be2ac11578a5aa6df2ba3b8724db3d
I20260925 10:33:26.633069 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:26.622604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42537 (local address 127.0.97.62:35091)
0925 10:33:26.623054 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:26.623059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:26.624252 (+ 1193us) server_negotiation.cc:407] Connection header received
0925 10:33:26.625955 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:26.625960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:26.626045 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:26.626157 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:26.627236 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:26.627973 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:26.629538 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:26.629991 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:26.630121 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:26.631611 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:26.631617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:26.631621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:33:26.631646 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:26.631685 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:33:26.631697 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:26.631792 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:26.632643 (+ 851us) server_negotiation.cc:299] Negotiation successful
0925 10:33:26.632721 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":258,"threads_started":1}
I20260925 10:33:26.633734 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:26.623162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35091 (local address 127.0.97.1:42537)
0925 10:33:26.624097 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:26.624139 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:26.625677 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:26.626429 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:26.626436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:26.626519 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:26.627073 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:26.627090 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:26.628144 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:26.628148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:26.629385 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:26.629397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:26.630232 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:26.630239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:26.630577 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:26.631316 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:33:26.631319 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:26.631357 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:33:26.631432 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:26.631791 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:26.631803 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:26.631812 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:33:26.631998 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 10:33:26.632362 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":299,"threads_started":1}
I20260925 10:33:26.633798 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35091
--builtin_ntp_servers=127.0.97.20:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 10:33:26.635279 9903 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35091
I20260925 10:33:26.635572 9903 heartbeater.cc:464] Registering TS with master...
I20260925 10:33:26.636318 9903 heartbeater.cc:511] Master 127.0.97.62:35091 requested a full tablet report, sending...
I20260925 10:33:26.638435 9717 ts_manager.cc:194] Registered new tserver with Master: 3b45968edbee4e36b85b931a30e9732d (127.0.97.1:32917)
I20260925 10:33:26.640528 9717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:42537
W20260925 10:33:26.954447 9907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:26.954689 9907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:26.954843 9907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:26.961894 9907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:26.961982 9907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:26.962137 9907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:33:26.971025 9907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:26.972190 9907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:26.973944 9907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:26.986289 9912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:26.986356 9913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:26.986389 9915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:26.987075 9907 server_base.cc:1034] running on GCE node
I20260925 10:33:26.987603 9907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:26.988437 9907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:26.989666 9907 hybrid_clock.cc:648] HybridClock initialized: now 1790332406989629 us; error 49 us; skew 500 ppm
I20260925 10:33:26.989976 9907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:33:26.992218 9907 webserver.cc:460] Webserver started at http://127.0.97.2:38479/ using document root <none> and password file <none>
I20260925 10:33:26.992872 9907 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:26.993000 9907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:26.993305 9907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:26.996076 9907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "8ad227782dae48b8853038b49ccd25a1"
format_stamp: "Formatted at 2026-09-25 10:33:26 on dist-test-slave-dxdd"
server_key: "91a3bba97888be0b68db9b3e4d341f45"
server_key_iv: "5b73ef699129accba9c36cc167c27e70"
server_key_version: "encryptionkey@0"
I20260925 10:33:26.996871 9907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8ad227782dae48b8853038b49ccd25a1"
format_stamp: "Formatted at 2026-09-25 10:33:26 on dist-test-slave-dxdd"
server_key: "91a3bba97888be0b68db9b3e4d341f45"
server_key_iv: "5b73ef699129accba9c36cc167c27e70"
server_key_version: "encryptionkey@0"
I20260925 10:33:27.001561 9907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 10:33:27.004771 9921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:27.005946 9907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:33:27.006107 9907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "8ad227782dae48b8853038b49ccd25a1"
format_stamp: "Formatted at 2026-09-25 10:33:26 on dist-test-slave-dxdd"
server_key: "91a3bba97888be0b68db9b3e4d341f45"
server_key_iv: "5b73ef699129accba9c36cc167c27e70"
server_key_version: "encryptionkey@0"
I20260925 10:33:27.006302 9907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:27.042074 9907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:27.043067 9907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:27.043370 9907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:27.044327 9907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:33:27.049089 9907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:27.049206 9907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:27.049319 9907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:27.049410 9907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:27.086898 9907 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41283
I20260925 10:33:27.086971 10033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41283 every 8 connection(s)
I20260925 10:33:27.088222 9907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:33:27.095898 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 9907
I20260925 10:33:27.096060 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:33:27.096508 388 external_mini_cluster.cc:1428] Setting key bb89918352a2942142f1b114671e356f
I20260925 10:33:27.099162 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35091
--builtin_ntp_servers=127.0.97.20:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 10:33:27.102053 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:27.090905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43747 (local address 127.0.97.62:35091)
0925 10:33:27.091095 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:27.091101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:27.092596 (+ 1495us) server_negotiation.cc:407] Connection header received
0925 10:33:27.094199 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:27.094205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:27.094276 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:27.094397 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:27.095384 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:27.096505 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:27.098828 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:27.099511 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:27.099637 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:27.101121 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:27.101129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:27.101136 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:33:27.101158 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:27.101196 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:33:27.101213 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:27.101308 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:27.101527 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 10:33:27.101654 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:33:27.103215 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:27.091422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35091 (local address 127.0.97.2:43747)
0925 10:33:27.092432 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:27.092473 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:27.093963 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:27.094601 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:27.094609 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:27.094695 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:27.095245 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:27.095259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:27.096718 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:27.096725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:27.098682 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:27.098696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:27.099733 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:27.099740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:27.100032 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:27.100800 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:33:27.100804 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:27.100849 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:33:27.100935 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:27.101319 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:27.101328 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:27.101336 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 10:33:27.101503 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 10:33:27.101834 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":342,"threads_started":1}
I20260925 10:33:27.104641 10034 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35091
I20260925 10:33:27.104951 10034 heartbeater.cc:464] Registering TS with master...
I20260925 10:33:27.105810 10034 heartbeater.cc:511] Master 127.0.97.62:35091 requested a full tablet report, sending...
I20260925 10:33:27.107203 9717 ts_manager.cc:194] Registered new tserver with Master: 8ad227782dae48b8853038b49ccd25a1 (127.0.97.2:41283)
I20260925 10:33:27.108007 9717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:43747
W20260925 10:33:27.413103 10037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:27.413353 10037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:27.413432 10037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:27.420336 10037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:27.420430 10037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:27.420531 10037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:33:27.429153 10037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:27.430434 10037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:27.432137 10037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:27.444818 10042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:27.444728 10043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:27.444763 10045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:27.445501 10037 server_base.cc:1034] running on GCE node
I20260925 10:33:27.445989 10037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:27.446825 10037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:27.448014 10037 hybrid_clock.cc:648] HybridClock initialized: now 1790332407447966 us; error 41 us; skew 500 ppm
I20260925 10:33:27.448324 10037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:33:27.456729 10037 webserver.cc:460] Webserver started at http://127.0.97.3:38689/ using document root <none> and password file <none>
I20260925 10:33:27.457372 10037 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:27.457495 10037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:27.457813 10037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:27.460563 10037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "7f87c020f6294dc0a3f4d2b35cdca780"
format_stamp: "Formatted at 2026-09-25 10:33:27 on dist-test-slave-dxdd"
server_key: "5ee7a1f6848d563cefeaa76cd5e980c3"
server_key_iv: "fda820b4c7a958284d50e47fdeb6807a"
server_key_version: "encryptionkey@0"
I20260925 10:33:27.461305 10037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7f87c020f6294dc0a3f4d2b35cdca780"
format_stamp: "Formatted at 2026-09-25 10:33:27 on dist-test-slave-dxdd"
server_key: "5ee7a1f6848d563cefeaa76cd5e980c3"
server_key_iv: "fda820b4c7a958284d50e47fdeb6807a"
server_key_version: "encryptionkey@0"
I20260925 10:33:27.466070 10037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260925 10:33:27.469483 10051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:27.470777 10037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:33:27.470928 10037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "7f87c020f6294dc0a3f4d2b35cdca780"
format_stamp: "Formatted at 2026-09-25 10:33:27 on dist-test-slave-dxdd"
server_key: "5ee7a1f6848d563cefeaa76cd5e980c3"
server_key_iv: "fda820b4c7a958284d50e47fdeb6807a"
server_key_version: "encryptionkey@0"
I20260925 10:33:27.471131 10037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:27.492648 10037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:27.493427 10037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:27.493731 10037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:27.494620 10037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:33:27.496029 10037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:27.496138 10037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:27.496264 10037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:27.496357 10037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:27.527765 10037 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46629
I20260925 10:33:27.527856 10163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46629 every 8 connection(s)
I20260925 10:33:27.528990 10037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:33:27.531718 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 10037
I20260925 10:33:27.531883 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:33:27.532311 388 external_mini_cluster.cc:1428] Setting key 74cd8bdcaea77c16c5c08d46ffc3aae9
I20260925 10:33:27.541105 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:27.531418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37207 (local address 127.0.97.62:35091)
0925 10:33:27.531624 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:27.531629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:27.533229 (+ 1600us) server_negotiation.cc:407] Connection header received
0925 10:33:27.535287 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:27.535292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:27.535383 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:27.535495 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:27.536464 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:27.537181 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:27.538752 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:27.539153 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:27.539253 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:27.540512 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:27.540516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:27.540519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:33:27.540534 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:27.540560 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:33:27.540570 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:27.540636 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:27.540809 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 10:33:27.540874 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:33:27.541956 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:27.532061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35091 (local address 127.0.97.3:37207)
0925 10:33:27.533095 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:27.533143 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:27.535010 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:27.535703 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:27.535711 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:27.535792 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:27.536306 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:27.536321 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:27.537329 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:27.537335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:27.538619 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:27.538636 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:27.539333 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:27.539338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:27.539539 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:27.540267 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:33:27.540269 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:27.540303 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:33:27.540372 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:27.540650 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:27.540656 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:27.540661 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:33:27.540788 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 10:33:27.541048 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":332,"threads_started":1}
I20260925 10:33:27.543076 10164 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35091
I20260925 10:33:27.543339 10164 heartbeater.cc:464] Registering TS with master...
I20260925 10:33:27.543982 10164 heartbeater.cc:511] Master 127.0.97.62:35091 requested a full tablet report, sending...
I20260925 10:33:27.545166 9717 ts_manager.cc:194] Registered new tserver with Master: 7f87c020f6294dc0a3f4d2b35cdca780 (127.0.97.3:46629)
I20260925 10:33:27.546193 9717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:37207
I20260925 10:33:27.547335 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 10:33:27.552932 9905 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:27.551428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.0.97.62:35091)
0925 10:33:27.551590 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:27.551594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:27.552091 (+ 497us) server_negotiation.cc:407] Connection header received
0925 10:33:27.552210 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:27.552213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:27.552275 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:27.552350 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:27.552685 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33734: BlockingRecv error: recv got EOF from 127.0.0.1:33734 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:33:27.553692 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 9776
I20260925 10:33:27.560819 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:27.554610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.0.97.1:32917)
0925 10:33:27.555136 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:27.555142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:27.555166 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:33:27.555216 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:27.555221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:27.555405 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:27.555523 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:27.555829 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:27.556868 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:27.558030 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:27.558620 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:27.558712 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:27.559584 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:27.559591 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:27.559602 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:33:27.560209 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:27.560250 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:33:27.560268 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:27.560354 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:27.560445 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 10:33:27.560553 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":270,"threads_started":1}
I20260925 10:33:27.561932 9898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:27.642253 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 9776
I20260925 10:33:27.658882 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 9907
I20260925 10:33:27.666285 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:27.659730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.0.97.2:41283)
0925 10:33:27.660253 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:27.660260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:27.660287 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:33:27.660343 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:27.660350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:27.660572 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:27.660697 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:27.661057 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:27.662038 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:27.663227 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:27.663770 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:27.663863 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:27.664891 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:27.664899 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:27.664913 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:33:27.665498 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:27.665541 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:33:27.665558 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:27.665677 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:27.665835 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 10:33:27.665954 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":268,"threads_started":1}
I20260925 10:33:27.667554 10029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:27.748227 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 9907
I20260925 10:33:27.764036 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 10037
I20260925 10:33:27.771679 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:27.765012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.0.97.3:46629)
0925 10:33:27.765630 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:27.765637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:27.765666 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:33:27.765723 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:27.765729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:27.765946 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:27.766073 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:27.766544 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:27.767563 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:27.768730 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:27.769236 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:27.769325 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:27.770248 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:27.770256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:27.770268 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:33:27.770884 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:27.770927 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:33:27.770945 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:27.771036 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:27.771182 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 10:33:27.771311 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":314,"threads_started":1}
I20260925 10:33:27.772850 10159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:27.852622 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 10037
I20260925 10:33:27.868369 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 9686
I20260925 10:33:27.869522 9747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:27.971398 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 9686
2026-09-25T10:33:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2359 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T10:33:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:33:28Z Disabled control of system clock
I20260925 10:33:28.039001 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45861
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:33:28.352602 10183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:28.352888 10183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:28.352972 10183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:28.360261 10183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:33:28.360366 10183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:28.360404 10183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:28.360471 10183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:33:28.360512 10183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:33:28.369822 10183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:28.371104 10183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:28.372803 10183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:28.384335 10188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:28.384307 10191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:28.384392 10189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:28.385074 10183 server_base.cc:1034] running on GCE node
I20260925 10:33:28.385833 10183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:28.387166 10183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:28.388363 10183 hybrid_clock.cc:648] HybridClock initialized: now 1790332408388318 us; error 35 us; skew 500 ppm
I20260925 10:33:28.388725 10183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:33:28.391076 10183 webserver.cc:460] Webserver started at http://127.0.97.62:39909/ using document root <none> and password file <none>
I20260925 10:33:28.391774 10183 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:28.391882 10183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:28.392215 10183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:28.395054 10183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "63b81e6cf3504054b120f164e141f330"
format_stamp: "Formatted at 2026-09-25 10:33:28 on dist-test-slave-dxdd"
server_key: "d4aabc68a3a70826ca47c9ce69623e72"
server_key_iv: "5032bba38babb3f37c5e78cf784853c6"
server_key_version: "encryptionkey@0"
I20260925 10:33:28.395805 10183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "63b81e6cf3504054b120f164e141f330"
format_stamp: "Formatted at 2026-09-25 10:33:28 on dist-test-slave-dxdd"
server_key: "d4aabc68a3a70826ca47c9ce69623e72"
server_key_iv: "5032bba38babb3f37c5e78cf784853c6"
server_key_version: "encryptionkey@0"
I20260925 10:33:28.400740 10183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:33:28.404021 10197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:28.405256 10183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:33:28.405419 10183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "63b81e6cf3504054b120f164e141f330"
format_stamp: "Formatted at 2026-09-25 10:33:28 on dist-test-slave-dxdd"
server_key: "d4aabc68a3a70826ca47c9ce69623e72"
server_key_iv: "5032bba38babb3f37c5e78cf784853c6"
server_key_version: "encryptionkey@0"
I20260925 10:33:28.405630 10183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:28.411223 10183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:28.412001 10183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:28.412282 10183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:28.429209 10183 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45861
I20260925 10:33:28.429243 10248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45861 every 8 connection(s)
I20260925 10:33:28.430577 10183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:33:28.434231 10249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:28.438525 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 10183
I20260925 10:33:28.438732 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:33:28.439363 388 external_mini_cluster.cc:1428] Setting key fe809642898d220ce06de3e443481458
I20260925 10:33:28.445801 10249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330: Bootstrap starting.
I20260925 10:33:28.448589 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:28.441327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43134 (local address 127.0.97.62:45861)
0925 10:33:28.442443 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:28.442461 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:28.442513 (+ 52us) server_negotiation.cc:407] Connection header received
0925 10:33:28.444795 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:28.444823 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:28.445303 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:28.445723 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:28.446701 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:28.446709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:28.446725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:33:28.446747 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:28.446823 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:33:28.446841 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:28.447049 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:28.447166 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 10:33:28.447434 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":361,"threads_started":1}
I20260925 10:33:28.449435 10249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:28.450775 10249 log.cc:824] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330: Log is configured to *not* fsync() on all Append() calls
I20260925 10:33:28.453347 10249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330: No bootstrap required, opened a new log
I20260925 10:33:28.460708 10249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b81e6cf3504054b120f164e141f330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45861 } }
I20260925 10:33:28.461118 10249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:28.461202 10249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b81e6cf3504054b120f164e141f330, State: Initialized, Role: FOLLOWER
I20260925 10:33:28.461660 10249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b81e6cf3504054b120f164e141f330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45861 } }
I20260925 10:33:28.461858 10249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:33:28.461944 10249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:33:28.462050 10249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:28.464102 10249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b81e6cf3504054b120f164e141f330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45861 } }
I20260925 10:33:28.464525 10249 leader_election.cc:302] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63b81e6cf3504054b120f164e141f330; no voters:
I20260925 10:33:28.465011 10249 leader_election.cc:288] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:33:28.465138 10254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:33:28.465890 10254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [term 1 LEADER]: Becoming Leader. State: Replica: 63b81e6cf3504054b120f164e141f330, State: Running, Role: LEADER
I20260925 10:33:28.466466 10254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b81e6cf3504054b120f164e141f330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45861 } }
I20260925 10:33:28.467319 10249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:33:28.469281 10256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63b81e6cf3504054b120f164e141f330. Latest consensus state: current_term: 1 leader_uuid: "63b81e6cf3504054b120f164e141f330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b81e6cf3504054b120f164e141f330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45861 } } }
I20260925 10:33:28.469729 10256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [sys.catalog]: This master's current role is: LEADER
I20260925 10:33:28.470557 10255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63b81e6cf3504054b120f164e141f330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b81e6cf3504054b120f164e141f330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45861 } } }
I20260925 10:33:28.470948 10255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330 [sys.catalog]: This master's current role is: LEADER
I20260925 10:33:28.474543 10263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:33:28.479411 10263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:33:28.486739 10263 catalog_manager.cc:1347] Generated new cluster ID: 771cf89d1351457d8c9e1ae032de9f86
I20260925 10:33:28.486850 10263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:33:28.500162 10263 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:33:28.500624 10263 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:33:28.509387 10263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63b81e6cf3504054b120f164e141f330: Generated new TSK 0
I20260925 10:33:28.510412 10263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:33:28.525266 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45861
--builtin_ntp_servers=127.0.97.20:36033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:33:28.843204 10273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:28.843534 10273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:28.843709 10273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:28.850816 10273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:28.850968 10273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:28.851173 10273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:33:28.860224 10273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:28.861541 10273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:28.863291 10273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:28.875826 10279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:28.875903 10281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:28.875880 10278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:28.876930 10273 server_base.cc:1034] running on GCE node
I20260925 10:33:28.877467 10273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:28.878389 10273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:28.879614 10273 hybrid_clock.cc:648] HybridClock initialized: now 1790332408879539 us; error 63 us; skew 500 ppm
I20260925 10:33:28.879909 10273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:33:28.882022 10273 webserver.cc:460] Webserver started at http://127.0.97.1:42629/ using document root <none> and password file <none>
I20260925 10:33:28.882726 10273 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:28.882850 10273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:28.883144 10273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:28.885811 10273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "153ca123580d4c758b638ab011f962c4"
format_stamp: "Formatted at 2026-09-25 10:33:28 on dist-test-slave-dxdd"
server_key: "e8d7da47a88e094857ef972ba32a7bec"
server_key_iv: "c86bc619e35b2918dc0782693a6cf0ce"
server_key_version: "encryptionkey@0"
I20260925 10:33:28.886623 10273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "153ca123580d4c758b638ab011f962c4"
format_stamp: "Formatted at 2026-09-25 10:33:28 on dist-test-slave-dxdd"
server_key: "e8d7da47a88e094857ef972ba32a7bec"
server_key_iv: "c86bc619e35b2918dc0782693a6cf0ce"
server_key_version: "encryptionkey@0"
I20260925 10:33:28.891275 10273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:33:28.894972 10287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:28.896344 10273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:33:28.896459 10273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "153ca123580d4c758b638ab011f962c4"
format_stamp: "Formatted at 2026-09-25 10:33:28 on dist-test-slave-dxdd"
server_key: "e8d7da47a88e094857ef972ba32a7bec"
server_key_iv: "c86bc619e35b2918dc0782693a6cf0ce"
server_key_version: "encryptionkey@0"
I20260925 10:33:28.896696 10273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:28.902102 10273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:28.902941 10273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:28.903152 10273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:28.904078 10273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:33:28.905367 10273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:28.905447 10273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:28.905515 10273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:28.905570 10273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:28.936067 10273 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41299
I20260925 10:33:28.936143 10399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41299 every 8 connection(s)
I20260925 10:33:28.937331 10273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:33:28.944826 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:28.939727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59787 (local address 127.0.97.62:45861)
0925 10:33:28.939887 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:28.939892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:28.941081 (+ 1189us) server_negotiation.cc:407] Connection header received
0925 10:33:28.942703 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:28.942709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:28.942773 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:28.942884 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:28.944098 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:28.944103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:28.944108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:33:28.944132 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:28.944161 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:33:28.944171 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:28.944290 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:28.944477 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 10:33:28.944548 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:33:28.945837 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 10273
I20260925 10:33:28.945868 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:28.940089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45861 (local address 127.0.97.1:59787)
0925 10:33:28.940933 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:28.940970 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:28.942462 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:28.943119 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:28.943127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:28.943210 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:28.943832 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:33:28.943834 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:28.943868 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:33:28.943941 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:28.944274 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:28.944284 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:28.944291 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:33:28.944556 (+ 265us) client_negotiation.cc:240] Negotiation successful
0925 10:33:28.944789 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":266,"threads_started":1}
I20260925 10:33:28.946056 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:33:28.946817 388 external_mini_cluster.cc:1428] Setting key c2fdf06d82a423627dc5bd01890051c6
I20260925 10:33:28.947338 10400 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45861
I20260925 10:33:28.947626 10400 heartbeater.cc:464] Registering TS with master...
I20260925 10:33:28.948241 10400 heartbeater.cc:511] Master 127.0.97.62:45861 requested a full tablet report, sending...
I20260925 10:33:28.949626 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45861
--builtin_ntp_servers=127.0.97.20:36033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 10:33:28.950364 10214 ts_manager.cc:194] Registered new tserver with Master: 153ca123580d4c758b638ab011f962c4 (127.0.97.1:41299)
W20260925 10:33:29.269484 10403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:29.269752 10403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:29.269847 10403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:29.277016 10403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:29.277107 10403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:29.277220 10403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:33:29.286079 10403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:29.287293 10403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:29.289011 10403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:29.301591 10408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:29.301548 10409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:29.301652 10411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:29.302650 10403 server_base.cc:1034] running on GCE node
I20260925 10:33:29.303198 10403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:29.304044 10403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:29.305258 10403 hybrid_clock.cc:648] HybridClock initialized: now 1790332409305214 us; error 57 us; skew 500 ppm
I20260925 10:33:29.305559 10403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:33:29.307699 10403 webserver.cc:460] Webserver started at http://127.0.97.2:43841/ using document root <none> and password file <none>
I20260925 10:33:29.308347 10403 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:29.308477 10403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:29.308779 10403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:29.313169 10403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "37f7e671adb34afa93c714fed2d148ae"
format_stamp: "Formatted at 2026-09-25 10:33:29 on dist-test-slave-dxdd"
server_key: "a96b91578caea2f1a8a36639fc5d599c"
server_key_iv: "60430a529252bf82d8eb29740005842c"
server_key_version: "encryptionkey@0"
I20260925 10:33:29.313939 10403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "37f7e671adb34afa93c714fed2d148ae"
format_stamp: "Formatted at 2026-09-25 10:33:29 on dist-test-slave-dxdd"
server_key: "a96b91578caea2f1a8a36639fc5d599c"
server_key_iv: "60430a529252bf82d8eb29740005842c"
server_key_version: "encryptionkey@0"
I20260925 10:33:29.318856 10403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 10:33:29.322093 10417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:29.323364 10403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:33:29.323527 10403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "37f7e671adb34afa93c714fed2d148ae"
format_stamp: "Formatted at 2026-09-25 10:33:29 on dist-test-slave-dxdd"
server_key: "a96b91578caea2f1a8a36639fc5d599c"
server_key_iv: "60430a529252bf82d8eb29740005842c"
server_key_version: "encryptionkey@0"
I20260925 10:33:29.323760 10403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:29.329136 10403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:29.329841 10403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:29.330119 10403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:29.331066 10403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:33:29.332388 10403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:29.332512 10403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:29.332644 10403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:29.332732 10403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:29.364276 10403 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37141
I20260925 10:33:29.364337 10529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37141 every 8 connection(s)
I20260925 10:33:29.365576 10403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:33:29.370257 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 10403
I20260925 10:33:29.370503 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:33:29.370916 388 external_mini_cluster.cc:1428] Setting key 8341bb7da68488db82894c13d67773b6
I20260925 10:33:29.373899 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45861
--builtin_ntp_servers=127.0.97.20:36033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 10:33:29.374096 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:29.368177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48801 (local address 127.0.97.62:45861)
0925 10:33:29.368357 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:29.368361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:29.369755 (+ 1394us) server_negotiation.cc:407] Connection header received
0925 10:33:29.371709 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:29.371714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:29.371776 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:29.371883 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:29.373324 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:29.373329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:29.373334 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:33:29.373354 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:29.373391 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:33:29.373405 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:29.373531 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:29.373695 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 10:33:29.373755 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:33:29.375262 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:29.368711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45861 (local address 127.0.97.2:48801)
0925 10:33:29.369595 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:29.369631 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:29.371402 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:29.372165 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:29.372176 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:29.372296 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:29.373025 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:33:29.373027 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:29.373067 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:33:29.373152 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:29.373505 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:29.373512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:29.373517 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:33:29.373722 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 10:33:29.373930 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":270,"threads_started":1}
I20260925 10:33:29.379312 10530 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45861
I20260925 10:33:29.379613 10530 heartbeater.cc:464] Registering TS with master...
I20260925 10:33:29.380230 10530 heartbeater.cc:511] Master 127.0.97.62:45861 requested a full tablet report, sending...
I20260925 10:33:29.381575 10214 ts_manager.cc:194] Registered new tserver with Master: 37f7e671adb34afa93c714fed2d148ae (127.0.97.2:37141)
W20260925 10:33:29.686165 10533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:29.686436 10533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:29.686524 10533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:29.693423 10533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:29.693506 10533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:29.693606 10533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:33:29.702172 10533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:29.703413 10533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:29.705040 10533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:29.717828 10538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:29.717797 10541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:29.717868 10539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:29.718640 10533 server_base.cc:1034] running on GCE node
I20260925 10:33:29.719193 10533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:29.720041 10533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:29.721226 10533 hybrid_clock.cc:648] HybridClock initialized: now 1790332409721189 us; error 32 us; skew 500 ppm
I20260925 10:33:29.721527 10533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:33:29.723603 10533 webserver.cc:460] Webserver started at http://127.0.97.3:42653/ using document root <none> and password file <none>
I20260925 10:33:29.724233 10533 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:29.724347 10533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:29.724632 10533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:29.727291 10533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "d9169c194d564268bbcc117179c0b6ae"
format_stamp: "Formatted at 2026-09-25 10:33:29 on dist-test-slave-dxdd"
server_key: "185729a6e52b801a512d35a846caaccd"
server_key_iv: "0649d4dd96af0ba78576d4e130ecd926"
server_key_version: "encryptionkey@0"
I20260925 10:33:29.728009 10533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d9169c194d564268bbcc117179c0b6ae"
format_stamp: "Formatted at 2026-09-25 10:33:29 on dist-test-slave-dxdd"
server_key: "185729a6e52b801a512d35a846caaccd"
server_key_iv: "0649d4dd96af0ba78576d4e130ecd926"
server_key_version: "encryptionkey@0"
I20260925 10:33:29.732743 10533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:33:29.736070 10547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:29.737303 10533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:33:29.737465 10533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "d9169c194d564268bbcc117179c0b6ae"
format_stamp: "Formatted at 2026-09-25 10:33:29 on dist-test-slave-dxdd"
server_key: "185729a6e52b801a512d35a846caaccd"
server_key_iv: "0649d4dd96af0ba78576d4e130ecd926"
server_key_version: "encryptionkey@0"
I20260925 10:33:29.737679 10533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:29.742875 10533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:29.743583 10533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:29.743875 10533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:29.744706 10533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:33:29.745961 10533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:29.746070 10533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:29.746198 10533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:29.746295 10533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:29.777976 10533 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39959
I20260925 10:33:29.778023 10659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39959 every 8 connection(s)
I20260925 10:33:29.779373 10533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:33:29.784564 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 10533
I20260925 10:33:29.784756 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:33:29.785207 388 external_mini_cluster.cc:1428] Setting key 327d038ccf01aa307b071f826ce086e7
I20260925 10:33:29.787381 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:29.781838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42949 (local address 127.0.97.62:45861)
0925 10:33:29.782008 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:29.782012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:29.783397 (+ 1385us) server_negotiation.cc:407] Connection header received
0925 10:33:29.785094 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:29.785100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:29.785172 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:29.785279 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:29.786648 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:29.786655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:29.786661 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:33:29.786680 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:29.786714 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:33:29.786729 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:29.786868 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:29.787062 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 10:33:29.787124 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:33:29.788311 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:29.782400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45861 (local address 127.0.97.3:42949)
0925 10:33:29.783247 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:29.783281 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:29.784819 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:29.785513 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:29.785520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:29.785602 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:29.786311 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:33:29.786333 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:29.786386 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:33:29.786494 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:29.786840 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:29.786849 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:29.786857 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 10:33:29.787136 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 10:33:29.787377 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":268,"threads_started":1}
I20260925 10:33:29.789402 10660 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45861
I20260925 10:33:29.789671 10660 heartbeater.cc:464] Registering TS with master...
I20260925 10:33:29.790179 10660 heartbeater.cc:511] Master 127.0.97.62:45861 requested a full tablet report, sending...
I20260925 10:33:29.791534 10214 ts_manager.cc:194] Registered new tserver with Master: d9169c194d564268bbcc117179c0b6ae (127.0.97.3:39959)
I20260925 10:33:29.799891 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 10:33:29.805341 10252 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:33:29.804041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43142 (local address 127.0.97.62:45861)
0925 10:33:29.804180 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:29.804184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:29.804604 (+ 420us) server_negotiation.cc:407] Connection header received
0925 10:33:29.804704 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:29.804707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:29.804748 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:29.804818 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:29.805097 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43142: BlockingRecv error: recv got EOF from 127.0.0.1:43142 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:33:29.805927 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 10273
I20260925 10:33:29.809231 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:29.806943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56310 (local address 127.0.97.1:41299)
0925 10:33:29.807485 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:29.807490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:29.807512 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:33:29.807563 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:29.807568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:29.807752 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:29.807873 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:29.808652 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:29.808658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:29.808668 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:33:29.808685 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:29.808723 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:33:29.808755 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:29.808852 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:29.808945 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 10:33:29.809010 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":295,"threads_started":1}
I20260925 10:33:29.810276 10395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:29.889786 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 10273
I20260925 10:33:29.906672 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 10403
I20260925 10:33:29.910359 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:29.907514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.0.97.2:37141)
0925 10:33:29.908043 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:29.908050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:29.908073 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:33:29.908131 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:29.908137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:29.908375 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:29.908550 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:29.909598 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:29.909604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:29.909616 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:33:29.909632 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:29.909668 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:33:29.909682 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:29.909832 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:29.909933 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 10:33:29.910037 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":266,"threads_started":1}
I20260925 10:33:29.911535 10525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:29.991515 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 10403
I20260925 10:33:30.006973 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 10533
I20260925 10:33:30.011103 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:30.007925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.0.97.3:39959)
0925 10:33:30.008543 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:30.008557 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:30.008614 (+ 57us) server_negotiation.cc:407] Connection header received
0925 10:33:30.008725 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:30.008731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:30.009017 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:30.009181 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:30.010370 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:30.010379 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:30.010393 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:33:30.010416 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:30.010465 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:33:30.010479 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:30.010654 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:30.010722 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 10:33:30.010817 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":310,"threads_started":1}
I20260925 10:33:30.012291 10655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:30.092535 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 10533
I20260925 10:33:30.108090 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 10183
I20260925 10:33:30.109225 10244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:30.207847 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 10183
2026-09-25T10:33:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2236 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:33:30 dist-test-slave-dxdd krb5kdc[10678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:33:30 dist-test-slave-dxdd krb5kdc[10678](info): set up 2 sockets
Sep 25 10:33:30 dist-test-slave-dxdd krb5kdc[10678](info): commencing operation
krb5kdc: starting...
W20260925 10:33:31.307089 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:33:31 dist-test-slave-dxdd krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:33:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:33:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:33:31.529513 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38733
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:33:31.852185 10691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:31.852449 10691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:31.852535 10691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:31.859508 10691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:33:31.859606 10691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:31.859649 10691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:31.859712 10691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:33:31.859756 10691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:33:31.869195 10691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:31.870368 10691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:31.871546 10691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:31.882730 10696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:31.882766 10699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:31.882771 10697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:31.883558 10691 server_base.cc:1034] running on GCE node
I20260925 10:33:31.884241 10691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:31.885493 10691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:31.886682 10691 hybrid_clock.cc:648] HybridClock initialized: now 1790332411886624 us; error 50 us; skew 500 ppm
I20260925 10:33:31.887017 10691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:33:31 dist-test-slave-dxdd krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:33:31.889997 10691 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:33:31.891294 10691 webserver.cc:460] Webserver started at http://127.0.97.62:37891/ using document root <none> and password file <none>
I20260925 10:33:31.891870 10691 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:31.891963 10691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:31.892189 10691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:31.894897 10691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "70f43b22c67e4300b32942c8889688ee"
format_stamp: "Formatted at 2026-09-25 10:33:31 on dist-test-slave-dxdd"
server_key: "b6a09b5a4dd7233a8be45a564dde03d9"
server_key_iv: "874c69779f94d1652e855ab44ed421e4"
server_key_version: "encryptionkey@0"
I20260925 10:33:31.895562 10691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "70f43b22c67e4300b32942c8889688ee"
format_stamp: "Formatted at 2026-09-25 10:33:31 on dist-test-slave-dxdd"
server_key: "b6a09b5a4dd7233a8be45a564dde03d9"
server_key_iv: "874c69779f94d1652e855ab44ed421e4"
server_key_version: "encryptionkey@0"
I20260925 10:33:31.900206 10691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:33:31.903414 10706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:31.904651 10691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 10:33:31.904836 10691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "70f43b22c67e4300b32942c8889688ee"
format_stamp: "Formatted at 2026-09-25 10:33:31 on dist-test-slave-dxdd"
server_key: "b6a09b5a4dd7233a8be45a564dde03d9"
server_key_iv: "874c69779f94d1652e855ab44ed421e4"
server_key_version: "encryptionkey@0"
I20260925 10:33:31.905028 10691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:31.920203 10691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:31.924242 10691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:31.924512 10691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:31.942101 10691 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38733
I20260925 10:33:31.942133 10757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38733 every 8 connection(s)
I20260925 10:33:31.943389 10691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:33:31.946864 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:31.949857 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 10691
I20260925 10:33:31.950085 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:33:31.950767 388 external_mini_cluster.cc:1428] Setting key 9c8ab17067fd0910a1ce707c67f429f3
I20260925 10:33:31.959016 10758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee: Bootstrap starting.
I20260925 10:33:31.962656 10758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:31.963914 10758 log.cc:824] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:33:31 dist-test-slave-dxdd krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:33:31.966820 10758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee: No bootstrap required, opened a new log
I20260925 10:33:31.974592 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:31.952576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.0.97.62:38733)
0925 10:33:31.953294 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:31.953305 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:31.953356 (+ 51us) server_negotiation.cc:407] Connection header received
0925 10:33:31.954999 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:31.955019 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:31.955379 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:31.955684 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:31.957445 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:31.958763 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:31.960558 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:31.961228 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:31.961356 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:31.966265 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:31.966273 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:31.966285 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:31.966375 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:31.970237 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:31.970906 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:31.970914 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:31.970922 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:31.971118 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:31.971460 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:31.971464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:31.971468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:31.972102 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:31.972410 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:31.972805 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 10:33:31.973255 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":341,"threads_started":1}
I20260925 10:33:31.976778 10758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f43b22c67e4300b32942c8889688ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } }
I20260925 10:33:31.977146 10758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:31.977272 10758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f43b22c67e4300b32942c8889688ee, State: Initialized, Role: FOLLOWER
I20260925 10:33:31.977794 10758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f43b22c67e4300b32942c8889688ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } }
I20260925 10:33:31.978021 10758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:33:31.978166 10758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:33:31.978369 10758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:31.980499 10758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f43b22c67e4300b32942c8889688ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } }
I20260925 10:33:31.980955 10758 leader_election.cc:302] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70f43b22c67e4300b32942c8889688ee; no voters:
I20260925 10:33:31.981554 10758 leader_election.cc:288] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:33:31.981686 10763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:33:31.982451 10763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [term 1 LEADER]: Becoming Leader. State: Replica: 70f43b22c67e4300b32942c8889688ee, State: Running, Role: LEADER
I20260925 10:33:31.982982 10763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f43b22c67e4300b32942c8889688ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } }
I20260925 10:33:31.983664 10758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:33:31.985803 10764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70f43b22c67e4300b32942c8889688ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f43b22c67e4300b32942c8889688ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } }
I20260925 10:33:31.985782 10765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70f43b22c67e4300b32942c8889688ee. Latest consensus state: current_term: 1 leader_uuid: "70f43b22c67e4300b32942c8889688ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f43b22c67e4300b32942c8889688ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } }
I20260925 10:33:31.986219 10765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [sys.catalog]: This master's current role is: LEADER
I20260925 10:33:31.986219 10764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee [sys.catalog]: This master's current role is: LEADER
I20260925 10:33:31.987097 10768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:33:31.992270 10768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:33:32.002095 10768 catalog_manager.cc:1347] Generated new cluster ID: a593ff8e28f74f08a3bba26eaa5196c5
I20260925 10:33:32.002233 10768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:33:32.023183 10768 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:33:32.024405 10768 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:33:32.032678 10768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee: Generated new TSK 0
I20260925 10:33:32.033449 10768 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:33:32.089998 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38733
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:33:32.406817 10784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:32.407081 10784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:32.407173 10784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:32.414537 10784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:32.414649 10784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:32.414777 10784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:33:32.424382 10784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:32.425561 10784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:32.426791 10784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:32.439558 10792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:32.440048 10789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:32.440075 10790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:32.440500 10784 server_base.cc:1034] running on GCE node
I20260925 10:33:32.441152 10784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:32.442096 10784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:32.443316 10784 hybrid_clock.cc:648] HybridClock initialized: now 1790332412443277 us; error 57 us; skew 500 ppm
I20260925 10:33:32.443617 10784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:33:32 dist-test-slave-dxdd krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:33:32.446643 10784 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:33:32.448037 10784 webserver.cc:460] Webserver started at http://127.0.97.1:42239/ using document root <none> and password file <none>
I20260925 10:33:32.448733 10784 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:32.448856 10784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:32.449167 10784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:32.451967 10784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "9dfa6f9ee1a043bd97e7061e8a1a6a3f"
format_stamp: "Formatted at 2026-09-25 10:33:32 on dist-test-slave-dxdd"
server_key: "716ffc6020cdf90dbd81cd913fbadb8f"
server_key_iv: "699993bcb9f64b6737abb3df772f4404"
server_key_version: "encryptionkey@0"
I20260925 10:33:32.452680 10784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9dfa6f9ee1a043bd97e7061e8a1a6a3f"
format_stamp: "Formatted at 2026-09-25 10:33:32 on dist-test-slave-dxdd"
server_key: "716ffc6020cdf90dbd81cd913fbadb8f"
server_key_iv: "699993bcb9f64b6737abb3df772f4404"
server_key_version: "encryptionkey@0"
I20260925 10:33:32.457535 10784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 10:33:32.460778 10799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:32.462051 10784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:33:32.462206 10784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "9dfa6f9ee1a043bd97e7061e8a1a6a3f"
format_stamp: "Formatted at 2026-09-25 10:33:32 on dist-test-slave-dxdd"
server_key: "716ffc6020cdf90dbd81cd913fbadb8f"
server_key_iv: "699993bcb9f64b6737abb3df772f4404"
server_key_version: "encryptionkey@0"
I20260925 10:33:32.462471 10784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:32.503214 10784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:32.509263 10784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:32.509601 10784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:32.510934 10784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:33:32.512892 10784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:32.512995 10784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:32.513135 10784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:32.513206 10784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:32.545769 10784 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38453
I20260925 10:33:32.545811 10911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38453 every 8 connection(s)
I20260925 10:33:32.547053 10784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:33:32.550086 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 10784
I20260925 10:33:32.550343 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:33:32.550963 388 external_mini_cluster.cc:1428] Setting key 5b45d64a0ae7d32797abe7bb1590f1a5
Sep 25 10:33:32 dist-test-slave-dxdd krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:33:32.566700 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:32.549477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45499 (local address 127.0.97.62:38733)
0925 10:33:32.549963 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:32.549971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:32.551339 (+ 1368us) server_negotiation.cc:407] Connection header received
0925 10:33:32.553323 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:32.553329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:32.553407 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:32.553529 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:32.554980 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:32.555683 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:32.557204 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:32.557606 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:32.557752 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:32.561209 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:32.561214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:32.561217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:32.561247 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:32.563758 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:32.564442 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:32.564454 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:32.564456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:32.564564 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:32.564978 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:32.564983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:32.564986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:32.565263 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:32.565424 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:32.566077 (+ 653us) server_negotiation.cc:299] Negotiation successful
0925 10:33:32.566294 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":232,"threads_started":1}
I20260925 10:33:32.568964 10915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:32.550055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.1:45499)
0925 10:33:32.550897 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:32.550941 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:32.553073 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:32.553757 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:32.553765 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:32.554236 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:32.554827 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:32.554844 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:32.555835 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:32.555840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:32.557028 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:32.557039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:32.557827 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:32.557836 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:32.558065 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:32.558776 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:32.558797 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:32.561001 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:32.563905 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:32.563913 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:32.563928 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:32.564320 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:32.564678 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:32.564683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:32.564686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:32.564858 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:32.565395 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:32.565403 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:32.565719 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 10:33:32.565981 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 10:33:32.566505 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":368,"threads_started":1}
I20260925 10:33:32.570470 10912 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38733
I20260925 10:33:32.570804 10912 heartbeater.cc:464] Registering TS with master...
I20260925 10:33:32.571666 10912 heartbeater.cc:511] Master 127.0.97.62:38733 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260925 10:33:32.573762 10723 ts_manager.cc:194] Registered new tserver with Master: 9dfa6f9ee1a043bd97e7061e8a1a6a3f (127.0.97.1:38453)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260925 10:33:32.575429 10723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:33:32.599632 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38733
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:33:32.915532 10918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:32.915789 10918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:32.915880 10918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:32.923679 10918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:32.923796 10918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:32.923931 10918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:33:32.934775 10918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:32.936022 10918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:32.937252 10918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:32.949939 10924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:32.949990 10926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:32.950157 10923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:32.950874 10918 server_base.cc:1034] running on GCE node
I20260925 10:33:32.951517 10918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:32.952492 10918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:32.953708 10918 hybrid_clock.cc:648] HybridClock initialized: now 1790332412953640 us; error 59 us; skew 500 ppm
I20260925 10:33:32.954039 10918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:33:32 dist-test-slave-dxdd krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:33:32.957548 10918 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:33:32.959371 10918 webserver.cc:460] Webserver started at http://127.0.97.2:45983/ using document root <none> and password file <none>
I20260925 10:33:32.960034 10918 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:32.960187 10918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:32.960482 10918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:32.963291 10918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "e1b4546d072f4cd6897c589e076ee5e5"
format_stamp: "Formatted at 2026-09-25 10:33:32 on dist-test-slave-dxdd"
server_key: "3e84de91d62ffd28cfcf2b253dc646e6"
server_key_iv: "7024da30ac17b472381be59d48777d95"
server_key_version: "encryptionkey@0"
I20260925 10:33:32.964015 10918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e1b4546d072f4cd6897c589e076ee5e5"
format_stamp: "Formatted at 2026-09-25 10:33:32 on dist-test-slave-dxdd"
server_key: "3e84de91d62ffd28cfcf2b253dc646e6"
server_key_iv: "7024da30ac17b472381be59d48777d95"
server_key_version: "encryptionkey@0"
I20260925 10:33:32.969117 10918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 10:33:32.972384 10933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:32.973556 10918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:33:32.973721 10918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "e1b4546d072f4cd6897c589e076ee5e5"
format_stamp: "Formatted at 2026-09-25 10:33:32 on dist-test-slave-dxdd"
server_key: "3e84de91d62ffd28cfcf2b253dc646e6"
server_key_iv: "7024da30ac17b472381be59d48777d95"
server_key_version: "encryptionkey@0"
I20260925 10:33:32.973910 10918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:32.988052 10918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:32.991803 10918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:32.992097 10918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:32.993017 10918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:33:32.994277 10918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:32.994437 10918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:32.994587 10918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:32.994664 10918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:33.025544 10918 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33257
I20260925 10:33:33.025589 11045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33257 every 8 connection(s)
I20260925 10:33:33.026777 10918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:33:33.029531 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 10918
I20260925 10:33:33.029724 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:33:33.030265 388 external_mini_cluster.cc:1428] Setting key 14aef4bbfc05d702e5e5010f17ec6ccc
Sep 25 10:33:33 dist-test-slave-dxdd krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:33:33.047159 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.029170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54335 (local address 127.0.97.62:38733)
0925 10:33:33.030273 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:33.030281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:33.030745 (+ 464us) server_negotiation.cc:407] Connection header received
0925 10:33:33.032634 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:33.032640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:33.032719 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:33.032841 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:33.034217 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.034968 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.036476 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.036882 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.037013 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.040940 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:33.040944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:33.040948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:33.040978 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:33.043543 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:33.044449 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:33.044452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:33.044454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:33.044565 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:33.045028 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:33.045031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:33.045033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:33.045414 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:33.045570 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:33.046485 (+ 915us) server_negotiation.cc:299] Negotiation successful
0925 10:33:33.046731 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954}
I20260925 10:33:33.048399 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.029813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.2:54335)
0925 10:33:33.030595 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:33.030629 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:33.032389 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:33.033079 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:33.033086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:33.033534 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:33.034055 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:33.034071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.035121 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.035125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:33.036330 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:33.036340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.037092 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.037098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:33.037332 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.037994 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:33.038014 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:33.040721 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:33.043699 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:33.043709 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:33.043727 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:33.044338 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:33.044669 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:33.044676 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:33.044683 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:33.044923 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:33.045555 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:33.045566 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:33.046038 (+ 472us) client_negotiation.cc:769] Sending connection context
0925 10:33:33.046358 (+ 320us) client_negotiation.cc:240] Negotiation successful
0925 10:33:33.046898 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":401,"threads_started":1}
I20260925 10:33:33.050102 11046 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38733
I20260925 10:33:33.050520 11046 heartbeater.cc:464] Registering TS with master...
I20260925 10:33:33.051496 11046 heartbeater.cc:511] Master 127.0.97.62:38733 requested a full tablet report, sending...
I20260925 10:33:33.053097 10723 ts_manager.cc:194] Registered new tserver with Master: e1b4546d072f4cd6897c589e076ee5e5 (127.0.97.2:33257)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260925 10:33:33.053929 10723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:33:33.081990 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38733
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:33:33.400440 11051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:33:33.400692 11051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:33:33.400786 11051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:33:33.407553 11051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:33:33.407680 11051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:33:33.407793 11051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:33:33.417337 11051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:33:33.418628 11051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:33:33.419864 11051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:33:33.432652 11056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:33.432693 11057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:33:33.432723 11059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:33:33.433893 11051 server_base.cc:1034] running on GCE node
I20260925 10:33:33.434533 11051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:33:33.435387 11051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:33:33.436617 11051 hybrid_clock.cc:648] HybridClock initialized: now 1790332413436569 us; error 33 us; skew 500 ppm
I20260925 10:33:33.436923 11051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:33:33 dist-test-slave-dxdd krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:33:33.439977 11051 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:33:33.441339 11051 webserver.cc:460] Webserver started at http://127.0.97.3:41593/ using document root <none> and password file <none>
I20260925 10:33:33.441979 11051 fs_manager.cc:359] Metadata directory not provided
I20260925 10:33:33.442101 11051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:33:33.442466 11051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:33:33.445127 11051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "96e94ecd3bf04092b0d0b811e9657576"
format_stamp: "Formatted at 2026-09-25 10:33:33 on dist-test-slave-dxdd"
server_key: "895041d85b138500d86639e84b032357"
server_key_iv: "8a16651be566e2e2e92ff5312d7f8bff"
server_key_version: "encryptionkey@0"
I20260925 10:33:33.445842 11051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "96e94ecd3bf04092b0d0b811e9657576"
format_stamp: "Formatted at 2026-09-25 10:33:33 on dist-test-slave-dxdd"
server_key: "895041d85b138500d86639e84b032357"
server_key_iv: "8a16651be566e2e2e92ff5312d7f8bff"
server_key_version: "encryptionkey@0"
I20260925 10:33:33.450791 11051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260925 10:33:33.453902 11066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:33.455098 11051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:33:33.455258 11051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "96e94ecd3bf04092b0d0b811e9657576"
format_stamp: "Formatted at 2026-09-25 10:33:33 on dist-test-slave-dxdd"
server_key: "895041d85b138500d86639e84b032357"
server_key_iv: "8a16651be566e2e2e92ff5312d7f8bff"
server_key_version: "encryptionkey@0"
I20260925 10:33:33.455441 11051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:33:33.484618 11051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:33:33.488875 11051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:33:33.489162 11051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:33:33.490067 11051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:33:33.491387 11051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:33:33.491457 11051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:33.491621 11051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:33:33.491678 11051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:33:33.523602 11051 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44727
I20260925 10:33:33.523684 11178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44727 every 8 connection(s)
I20260925 10:33:33.524822 11051 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:33:33.532647 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 11051
I20260925 10:33:33.532826 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:33:33.533301 388 external_mini_cluster.cc:1428] Setting key a37a6bf27139af2af24c13c26129097d
Sep 25 10:33:33 dist-test-slave-dxdd krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:33:33.543543 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.527253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47747 (local address 127.0.97.62:38733)
0925 10:33:33.527430 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:33.527434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:33.528604 (+ 1170us) server_negotiation.cc:407] Connection header received
0925 10:33:33.530385 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:33.530391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:33.530479 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:33.530590 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:33.531962 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.532996 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.534635 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.535028 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.535174 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.538399 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:33.538409 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:33.538413 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:33.538443 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:33.540678 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:33.541371 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:33.541378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:33.541381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:33.541507 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:33.541912 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:33.541918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:33.541922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:33.542177 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:33.542355 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:33.542996 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 10:33:33.543197 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:33:33.544438 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.527759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.3:47747)
0925 10:33:33.528462 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:33.528501 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:33.530076 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:33.530805 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:33.530813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:33.531274 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:33:33.531798 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:33.531813 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.533141 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.533145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:33.534495 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:33.534507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.535243 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.535253 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:33.535501 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.536171 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:33:33.536192 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:33:33.538169 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:33:33.540844 (+ 2675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:33.540853 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:33.540868 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:33.541247 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:33.541611 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:33:33.541617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:33:33.541621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:33:33.541806 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:33:33.542332 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:33:33.542340 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:33:33.542641 (+ 301us) client_negotiation.cc:769] Sending connection context
0925 10:33:33.542892 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 10:33:33.543393 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":264,"threads_started":1}
I20260925 10:33:33.545773 11179 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38733
I20260925 10:33:33.546073 11179 heartbeater.cc:464] Registering TS with master...
I20260925 10:33:33.546938 11179 heartbeater.cc:511] Master 127.0.97.62:38733 requested a full tablet report, sending...
I20260925 10:33:33.548413 10723 ts_manager.cc:194] Registered new tserver with Master: 96e94ecd3bf04092b0d0b811e9657576 (127.0.97.3:44727)
I20260925 10:33:33.549460 10723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47747
I20260925 10:33:33.560228 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:33:33.575505 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.564371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42270 (local address 127.0.97.62:38733)
0925 10:33:33.564514 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:33.564518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:33.564981 (+ 463us) server_negotiation.cc:407] Connection header received
0925 10:33:33.565109 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:33.565113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:33.565167 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:33.565244 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:33.566596 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.567274 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.568401 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.568799 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.568920 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.571068 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:33.571072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:33.571075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:33.571102 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:33.573312 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:33.573895 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:33.573899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:33.573901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:33.574001 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:33.574287 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:33.574290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:33.574292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:33.574597 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:33.574768 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:33.575038 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 10:33:33.575173 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:33:33.578148 10912 heartbeater.cc:503] Master 127.0.97.62:38733 was elected leader, sending a full tablet report...
I20260925 10:33:33.579169 10723 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:33:33.586012 10723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:33:33.609635 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.597787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38453 (local address 127.0.0.1:39654)
0925 10:33:33.598566 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:33.598582 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:33.598686 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:33.599523 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:33.599533 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:33.599565 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:33:33.600209 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:33.600230 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.602684 (+ 2454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.602692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:33.605664 (+ 2972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:33.605682 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.608367 (+ 2685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.608373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:33.608564 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.608617 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 10:33:33.608740 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 10:33:33.609014 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":99,"thread_start_us":768,"threads_started":1}
I20260925 10:33:33.610754 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.597109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.0.97.1:38453)
0925 10:33:33.597744 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:33.597752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:33.598699 (+ 947us) server_negotiation.cc:407] Connection header received
0925 10:33:33.598809 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:33.598816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:33.599073 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:33.599258 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:33:33.600467 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.602491 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.605899 (+ 3408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.607883 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.608004 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.608244 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:33.610088 (+ 1844us) server_negotiation.cc:299] Negotiation successful
0925 10:33:33.610333 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":878,"threads_started":1}
I20260925 10:33:33.611197 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.596776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59016 (local address 127.0.97.2:33257)
0925 10:33:33.597474 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:33.597486 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:33.598287 (+ 801us) server_negotiation.cc:407] Connection header received
0925 10:33:33.598725 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:33.598736 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:33.599047 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:33.599292 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:33:33.601124 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.603224 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.606667 (+ 3443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.608600 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.608729 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.608962 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:33.610569 (+ 1607us) server_negotiation.cc:299] Negotiation successful
0925 10:33:33.610758 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":429,"threads_started":1}
I20260925 10:33:33.610406 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.596607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33257 (local address 127.0.0.1:59016)
0925 10:33:33.598132 (+ 1525us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:33.598179 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:33.598478 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:33.599630 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:33.599641 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:33.599697 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:33:33.600902 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:33.600918 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.603454 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.603465 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:33.606450 (+ 2985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:33.606469 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.609225 (+ 2756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.609234 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:33.609491 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.609540 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 10:33:33.609644 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:33:33.609823 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1296,"mutex_wait_us":1,"thread_start_us":1373,"threads_started":1}
I20260925 10:33:33.615597 10847 tablet_service.cc:1467] Processing CreateTablet for tablet e29cbcbfc82b46b19c12b82661b943f2 (DEFAULT_TABLE table=test-table [id=342f360088f7466d97f5b5582c4a7d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:33:33.616261 10981 tablet_service.cc:1467] Processing CreateTablet for tablet e29cbcbfc82b46b19c12b82661b943f2 (DEFAULT_TABLE table=test-table [id=342f360088f7466d97f5b5582c4a7d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:33:33.616886 10847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e29cbcbfc82b46b19c12b82661b943f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:33.617442 10981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e29cbcbfc82b46b19c12b82661b943f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:33.620709 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.598899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44727 (local address 127.0.0.1:55820)
0925 10:33:33.611236 (+ 12337us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:33.611260 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:33.611729 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:33.614504 (+ 2775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:33.614509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:33.614535 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:33:33.614682 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:33.614691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.616494 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.616499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:33.618289 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:33.618324 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.619990 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.619995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:33.620182 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.620217 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:33:33.620315 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 10:33:33.620444 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12212,"thread_start_us":13047,"threads_started":1}
I20260925 10:33:33.620934 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.599265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.0.97.3:44727)
0925 10:33:33.613888 (+ 14623us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:33.613897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:33.613932 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:33:33.614036 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:33.614044 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:33.614280 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:33.614495 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:33:33.614849 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.616343 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.618462 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.619836 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.619972 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.620169 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:33.620362 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 10:33:33.620512 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14460,"thread_start_us":14297,"threads_started":1}
I20260925 10:33:33.624547 11114 tablet_service.cc:1467] Processing CreateTablet for tablet e29cbcbfc82b46b19c12b82661b943f2 (DEFAULT_TABLE table=test-table [id=342f360088f7466d97f5b5582c4a7d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:33:33.625914 11114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e29cbcbfc82b46b19c12b82661b943f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:33:33.629197 11197 tablet_bootstrap.cc:492] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f: Bootstrap starting.
I20260925 10:33:33.631064 11198 tablet_bootstrap.cc:492] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5: Bootstrap starting.
I20260925 10:33:33.632767 11197 tablet_bootstrap.cc:654] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:33.634047 11197 log.cc:824] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:33:33.634476 11198 tablet_bootstrap.cc:654] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:33.635846 11198 log.cc:824] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5: Log is configured to *not* fsync() on all Append() calls
I20260925 10:33:33.637723 11197 tablet_bootstrap.cc:492] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f: No bootstrap required, opened a new log
I20260925 10:33:33.638016 11197 ts_tablet_manager.cc:1397] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 10:33:33.638733 11198 tablet_bootstrap.cc:492] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5: No bootstrap required, opened a new log
I20260925 10:33:33.639030 11198 ts_tablet_manager.cc:1397] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260925 10:33:33.639142 11201 tablet_bootstrap.cc:492] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576: Bootstrap starting.
I20260925 10:33:33.641773 11201 tablet_bootstrap.cc:654] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576: Neither blocks nor log segments found. Creating new log.
I20260925 10:33:33.642741 11201 log.cc:824] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576: Log is configured to *not* fsync() on all Append() calls
I20260925 10:33:33.645156 11201 tablet_bootstrap.cc:492] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576: No bootstrap required, opened a new log
I20260925 10:33:33.645763 11201 ts_tablet_manager.cc:1397] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 10:33:33.648528 11197 raft_consensus.cc:348] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b4546d072f4cd6897c589e076ee5e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33257 } } peers { permanent_uuid: "9dfa6f9ee1a043bd97e7061e8a1a6a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38453 } } peers { permanent_uuid: "96e94ecd3bf04092b0d0b811e9657576" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44727 } }
I20260925 10:33:33.648955 11197 raft_consensus.cc:374] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:33.649094 11197 raft_consensus.cc:729] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfa6f9ee1a043bd97e7061e8a1a6a3f, State: Initialized, Role: FOLLOWER
I20260925 10:33:33.649576 11198 raft_consensus.cc:348] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b4546d072f4cd6897c589e076ee5e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33257 } } peers { permanent_uuid: "9dfa6f9ee1a043bd97e7061e8a1a6a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38453 } } peers { permanent_uuid: "96e94ecd3bf04092b0d0b811e9657576" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44727 } }
I20260925 10:33:33.649725 11197 consensus_queue.cc:260] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b4546d072f4cd6897c589e076ee5e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33257 } } peers { permanent_uuid: "9dfa6f9ee1a043bd97e7061e8a1a6a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38453 } } peers { permanent_uuid: "96e94ecd3bf04092b0d0b811e9657576" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44727 } }
I20260925 10:33:33.649989 11198 raft_consensus.cc:374] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:33.650085 11198 raft_consensus.cc:729] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1b4546d072f4cd6897c589e076ee5e5, State: Initialized, Role: FOLLOWER
I20260925 10:33:33.650818 11198 consensus_queue.cc:260] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b4546d072f4cd6897c589e076ee5e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33257 } } peers { permanent_uuid: "9dfa6f9ee1a043bd97e7061e8a1a6a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38453 } } peers { permanent_uuid: "96e94ecd3bf04092b0d0b811e9657576" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44727 } }
I20260925 10:33:33.651698 11197 ts_tablet_manager.cc:1428] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260925 10:33:33.651844 11046 heartbeater.cc:503] Master 127.0.97.62:38733 was elected leader, sending a full tablet report...
I20260925 10:33:33.653190 11198 ts_tablet_manager.cc:1428] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260925 10:33:33.656198 11201 raft_consensus.cc:348] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b4546d072f4cd6897c589e076ee5e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33257 } } peers { permanent_uuid: "9dfa6f9ee1a043bd97e7061e8a1a6a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38453 } } peers { permanent_uuid: "96e94ecd3bf04092b0d0b811e9657576" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44727 } }
I20260925 10:33:33.656525 11201 raft_consensus.cc:374] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:33:33.656625 11201 raft_consensus.cc:729] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e94ecd3bf04092b0d0b811e9657576, State: Initialized, Role: FOLLOWER
I20260925 10:33:33.657299 11201 consensus_queue.cc:260] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b4546d072f4cd6897c589e076ee5e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33257 } } peers { permanent_uuid: "9dfa6f9ee1a043bd97e7061e8a1a6a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38453 } } peers { permanent_uuid: "96e94ecd3bf04092b0d0b811e9657576" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44727 } }
I20260925 10:33:33.658269 11179 heartbeater.cc:503] Master 127.0.97.62:38733 was elected leader, sending a full tablet report...
I20260925 10:33:33.658939 11201 ts_tablet_manager.cc:1428] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260925 10:33:33.732290 11204 raft_consensus.cc:482] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:33:33.732555 11204 raft_consensus.cc:504] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b4546d072f4cd6897c589e076ee5e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33257 } } peers { permanent_uuid: "9dfa6f9ee1a043bd97e7061e8a1a6a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38453 } } peers { permanent_uuid: "96e94ecd3bf04092b0d0b811e9657576" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44727 } }
I20260925 10:33:33.734084 11204 leader_election.cc:288] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dfa6f9ee1a043bd97e7061e8a1a6a3f (127.0.97.1:38453), 96e94ecd3bf04092b0d0b811e9657576 (127.0.97.3:44727)
I20260925 10:33:33.742522 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.734568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55497 (local address 127.0.97.1:38453)
0925 10:33:33.734823 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:33.734830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:33.735144 (+ 314us) server_negotiation.cc:407] Connection header received
0925 10:33:33.735424 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:33.735431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:33.735522 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:33.735591 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:33:33.736412 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.737415 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.740039 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.741242 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.741362 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.741461 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:33.742126 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 10:33:33.742247 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:33:33.742517 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.735383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41643 (local address 127.0.97.3:44727)
0925 10:33:33.735580 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:33.735585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:33.735890 (+ 305us) server_negotiation.cc:407] Connection header received
0925 10:33:33.736097 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:33.736101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:33.736158 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:33.736233 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:33:33.736748 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.737721 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.740047 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.741229 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.741362 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.741462 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:33.742126 (+ 664us) server_negotiation.cc:299] Negotiation successful
0925 10:33:33.742247 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 10:33:33.742796 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.734387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38453 (local address 127.0.97.2:55497)
0925 10:33:33.735048 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:33.735068 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:33.735228 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:33.735798 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:33.735802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:33.735833 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:33:33.736242 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:33.736253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.737541 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.737545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:33.739865 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:33.739880 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.741530 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.741535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:33.741842 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.741988 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 10:33:33.742088 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 10:33:33.742276 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":184,"thread_start_us":329,"threads_started":1}
I20260925 10:33:33.742832 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.735044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44727 (local address 127.0.97.2:41643)
0925 10:33:33.735774 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:33.735794 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:33:33.735964 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:33:33.736265 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:33:33.736269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:33:33.736295 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:33:33.736590 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:33.736597 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.737848 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.737853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:33.739910 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:33:33.739923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.741526 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.741533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:33:33.741844 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.741989 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 10:33:33.742086 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 10:33:33.742276 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":167,"spinlock_wait_cycles":25472,"thread_start_us":405,"threads_started":1}
I20260925 10:33:33.743901 11134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29cbcbfc82b46b19c12b82661b943f2" candidate_uuid: "e1b4546d072f4cd6897c589e076ee5e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e94ecd3bf04092b0d0b811e9657576" is_pre_election: true
I20260925 10:33:33.743896 10867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29cbcbfc82b46b19c12b82661b943f2" candidate_uuid: "e1b4546d072f4cd6897c589e076ee5e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dfa6f9ee1a043bd97e7061e8a1a6a3f" is_pre_election: true
I20260925 10:33:33.744369 11134 raft_consensus.cc:2454] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1b4546d072f4cd6897c589e076ee5e5 in term 0.
I20260925 10:33:33.744359 10867 raft_consensus.cc:2454] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1b4546d072f4cd6897c589e076ee5e5 in term 0.
I20260925 10:33:33.744978 10934 leader_election.cc:302] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dfa6f9ee1a043bd97e7061e8a1a6a3f, e1b4546d072f4cd6897c589e076ee5e5; no voters:
I20260925 10:33:33.745359 11204 raft_consensus.cc:2781] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:33:33.745504 11204 raft_consensus.cc:482] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:33:33.745613 11204 raft_consensus.cc:3037] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:33.747792 11204 raft_consensus.cc:504] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b4546d072f4cd6897c589e076ee5e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33257 } } peers { permanent_uuid: "9dfa6f9ee1a043bd97e7061e8a1a6a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38453 } } peers { permanent_uuid: "96e94ecd3bf04092b0d0b811e9657576" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44727 } }
I20260925 10:33:33.748392 11204 leader_election.cc:288] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [CANDIDATE]: Term 1 election: Requested vote from peers 9dfa6f9ee1a043bd97e7061e8a1a6a3f (127.0.97.1:38453), 96e94ecd3bf04092b0d0b811e9657576 (127.0.97.3:44727)
I20260925 10:33:33.748745 10867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29cbcbfc82b46b19c12b82661b943f2" candidate_uuid: "e1b4546d072f4cd6897c589e076ee5e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dfa6f9ee1a043bd97e7061e8a1a6a3f"
I20260925 10:33:33.748862 11134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29cbcbfc82b46b19c12b82661b943f2" candidate_uuid: "e1b4546d072f4cd6897c589e076ee5e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e94ecd3bf04092b0d0b811e9657576"
I20260925 10:33:33.748958 10867 raft_consensus.cc:3037] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:33.749056 11134 raft_consensus.cc:3037] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:33:33.752117 11134 raft_consensus.cc:2454] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1b4546d072f4cd6897c589e076ee5e5 in term 1.
I20260925 10:33:33.752117 10867 raft_consensus.cc:2454] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1b4546d072f4cd6897c589e076ee5e5 in term 1.
I20260925 10:33:33.752491 10934 leader_election.cc:302] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dfa6f9ee1a043bd97e7061e8a1a6a3f, e1b4546d072f4cd6897c589e076ee5e5; no voters:
I20260925 10:33:33.752743 11204 raft_consensus.cc:2781] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:33:33.753165 11204 raft_consensus.cc:686] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [term 1 LEADER]: Becoming Leader. State: Replica: e1b4546d072f4cd6897c589e076ee5e5, State: Running, Role: LEADER
I20260925 10:33:33.753736 11204 consensus_queue.cc:237] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b4546d072f4cd6897c589e076ee5e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33257 } } peers { permanent_uuid: "9dfa6f9ee1a043bd97e7061e8a1a6a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38453 } } peers { permanent_uuid: "96e94ecd3bf04092b0d0b811e9657576" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44727 } }
I20260925 10:33:33.758117 10723 catalog_manager.cc:5495] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1b4546d072f4cd6897c589e076ee5e5 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e1b4546d072f4cd6897c589e076ee5e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b4546d072f4cd6897c589e076ee5e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dfa6f9ee1a043bd97e7061e8a1a6a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96e94ecd3bf04092b0d0b811e9657576" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44727 } health_report { overall_health: UNKNOWN } } }
W20260925 10:33:33.776335 11180 tablet.cc:2369] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:33:33.778540 11047 tablet.cc:2369] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:33:33.796171 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.790904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.0.97.2:33257)
0925 10:33:33.791102 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:33.791110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:33.791201 (+ 91us) server_negotiation.cc:407] Connection header received
0925 10:33:33.791436 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:33.791440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:33.791523 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:33.791681 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:33:33.792041 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.792777 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.794078 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.794526 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.794659 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.795129 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:33:33.795612 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:33:33.795765 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:33.795829 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 10:33:33.795902 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 10:33:33.799296 10913 tablet.cc:2369] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:33:33.804237 10867 raft_consensus.cc:1261] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f [term 1 FOLLOWER]: Refusing update from remote peer e1b4546d072f4cd6897c589e076ee5e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:33:33.804376 11134 raft_consensus.cc:1261] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576 [term 1 FOLLOWER]: Refusing update from remote peer e1b4546d072f4cd6897c589e076ee5e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:33:33.805153 11208 consensus_queue.cc:1035] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dfa6f9ee1a043bd97e7061e8a1a6a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:33:33.805538 11204 consensus_queue.cc:1035] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e94ecd3bf04092b0d0b811e9657576" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:33:33.816740 11211 mvcc.cc:205] Tried to move back new op lower bound from 7333201566932885504 to 7333201566739619840. Current Snapshot: MvccSnapshot[applied={T|T < 7333201566932885504}]
I20260925 10:33:33.832337 10723 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:33:33.832568 10723 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:33:33.835851 10723 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70f43b22c67e4300b32942c8889688ee: Sending DeleteTablet for 3 replicas of tablet e29cbcbfc82b46b19c12b82661b943f2
I20260925 10:33:33.837059 10981 tablet_service.cc:1514] Processing DeleteTablet for tablet e29cbcbfc82b46b19c12b82661b943f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:33:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59016
I20260925 10:33:33.837045 10847 tablet_service.cc:1514] Processing DeleteTablet for tablet e29cbcbfc82b46b19c12b82661b943f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:33:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39654
I20260925 10:33:33.837170 11114 tablet_service.cc:1514] Processing DeleteTablet for tablet e29cbcbfc82b46b19c12b82661b943f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:33:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55820
I20260925 10:33:33.838939 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 10784
I20260925 10:33:33.846989 11220 tablet_replica.cc:330] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5: stopping tablet replica
I20260925 10:33:33.848268 11220 raft_consensus.cc:2229] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:33:33.849781 11220 raft_consensus.cc:2258] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:33:33.853451 11220 ts_tablet_manager.cc:1905] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 10:33:33 dist-test-slave-dxdd krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:33:33.858973 11221 tablet_replica.cc:330] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f: stopping tablet replica
I20260925 10:33:33.859571 11222 tablet_replica.cc:330] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576: stopping tablet replica
I20260925 10:33:33.861150 11222 raft_consensus.cc:2229] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:33:33.861604 11221 raft_consensus.cc:2229] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:33:33.862152 11222 raft_consensus.cc:2258] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:33:33.862674 11221 raft_consensus.cc:2258] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:33:33.865000 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:33.840361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.0.97.1:38453)
0925 10:33:33.840530 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:33.840536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:33.845756 (+ 5220us) server_negotiation.cc:407] Connection header received
0925 10:33:33.846787 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:33.846794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:33.846897 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:33.847002 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:33.848653 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.849825 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.851904 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:33.852541 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:33.852636 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:33.857651 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:33.857667 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:33.857684 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:33.857724 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:33.861558 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:33.862434 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:33.862442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:33.862446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:33.862617 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:33.863081 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:33.863086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:33.863089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:33.863532 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:33.863802 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:33.864409 (+ 607us) server_negotiation.cc:299] Negotiation successful
0925 10:33:33.864614 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:33:33.865350 11221 ts_tablet_manager.cc:1905] T e29cbcbfc82b46b19c12b82661b943f2 P 9dfa6f9ee1a043bd97e7061e8a1a6a3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:33:33.865525 11220 ts_tablet_manager.cc:1918] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:33:33.865710 11220 log.cc:1196] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/e29cbcbfc82b46b19c12b82661b943f2
I20260925 10:33:33.866266 11220 ts_tablet_manager.cc:1939] T e29cbcbfc82b46b19c12b82661b943f2 P e1b4546d072f4cd6897c589e076ee5e5: Deleting consensus metadata
I20260925 10:33:33.867177 11222 ts_tablet_manager.cc:1905] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:33:33.868147 10709 catalog_manager.cc:4841] TS e1b4546d072f4cd6897c589e076ee5e5 (127.0.97.2:33257): tablet e29cbcbfc82b46b19c12b82661b943f2 (table test-table [id=342f360088f7466d97f5b5582c4a7d62]) successfully deleted
I20260925 10:33:33.866608 10907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:33.874457 11222 ts_tablet_manager.cc:1918] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:33:33.874610 11222 log.cc:1196] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/e29cbcbfc82b46b19c12b82661b943f2
I20260925 10:33:33.875037 11222 ts_tablet_manager.cc:1939] T e29cbcbfc82b46b19c12b82661b943f2 P 96e94ecd3bf04092b0d0b811e9657576: Deleting consensus metadata
I20260925 10:33:33.876350 10707 catalog_manager.cc:4841] TS 96e94ecd3bf04092b0d0b811e9657576 (127.0.97.3:44727): tablet e29cbcbfc82b46b19c12b82661b943f2 (table test-table [id=342f360088f7466d97f5b5582c4a7d62]) successfully deleted
I20260925 10:33:34.008329 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 10784
I20260925 10:33:34.025977 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 10918
W20260925 10:33:34.026376 10707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38453 (error 108)
I20260925 10:33:34.028638 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:34.028181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38453 (local address 127.0.0.1:39678)
0925 10:33:34.028364 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:34.028472 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 10:33:34.029381 10707 catalog_manager.cc:4584] TS 9dfa6f9ee1a043bd97e7061e8a1a6a3f (127.0.97.1:38453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e29cbcbfc82b46b19c12b82661b943f2: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38453: connect: Connection refused (error 111)
Sep 25 10:33:34 dist-test-slave-dxdd krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:33:34.047359 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:34.028526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.0.97.2:33257)
0925 10:33:34.028695 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:34.028700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:34.028719 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:33:34.028789 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:34.028794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:34.028874 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:34.029000 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:34.030645 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:34.031910 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:34.033697 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:34.034466 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:34.034600 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:34.039698 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:34.039711 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:34.039729 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:34.039773 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:34.043892 (+ 4119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:34.044715 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:34.044725 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:34.044729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:34.045204 (+ 475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:34.045624 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:34.045629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:34.045633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:34.046080 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:34.046410 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:34.046744 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 10:33:34.046963 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:33:34.048470 11041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:34.082947 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:34.082460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38453 (local address 127.0.0.1:39680)
0925 10:33:34.082704 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:34.082791 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 10:33:34.084004 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:34.083671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38453 (local address 127.0.0.1:39684)
0925 10:33:34.083791 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:34.083884 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 10:33:34.144863 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:34.144388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38453 (local address 127.0.0.1:39694)
0925 10:33:34.144626 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:34.144718 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 10:33:34.145880 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:34.145572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38453 (local address 127.0.0.1:39696)
0925 10:33:34.145697 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:34.145763 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 10:33:34.163128 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 10918
I20260925 10:33:34.182385 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 11051
Sep 25 10:33:34 dist-test-slave-dxdd krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:33:34.196794 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:34.183117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.0.97.3:44727)
0925 10:33:34.183274 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 10:33:34.183279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:33:34.183303 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:33:34.183457 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:33:34.183461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:33:34.183523 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:33:34.183607 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:33:34.185017 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:34.185753 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:34.187021 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:33:34.187388 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:33:34.187482 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:33:34.191582 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:33:34.191596 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:33:34.191612 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:33:34.191642 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:33:34.194173 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:34.195079 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:34.195087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:34.195089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:34.195239 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:33:34.195578 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:33:34.195581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:33:34.195583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:33:34.195926 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:33:34.196150 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 10:33:34.196380 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 10:33:34.196534 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:33:34.197484 11174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:34.226734 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:34.226196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38453 (local address 127.0.0.1:39706)
0925 10:33:34.226462 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:34.226569 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 10:33:34.227816 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:34.227370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38453 (local address 127.0.0.1:39710)
0925 10:33:34.227578 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:34.227660 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 10:33:34.286375 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 11051
I20260925 10:33:34.304885 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 10691
I20260925 10:33:34.306063 10753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:33:34.412323 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:33:34.411551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38453 (local address 127.0.0.1:39714)
0925 10:33:34.411822 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 10:33:34.411921 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260925 10:33:34.412366 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 10691
2026-09-25T10:33:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4203 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:33:34 dist-test-slave-dxdd krb5kdc[11231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:33:34 dist-test-slave-dxdd krb5kdc[11231](info): set up 2 sockets
Sep 25 10:33:34 dist-test-slave-dxdd krb5kdc[11231](info): commencing operation
krb5kdc: starting...
W20260925 10:33:35.511173 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:33:35 dist-test-slave-dxdd krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:33:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:33:35Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260925 10:33:35.816665 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4ieADW/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/postgres -l logfile start
2026-09-25 10:33:38.284 UTC [11261] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 10:33:38.284 UTC [11261] LOG: listening on IPv4 address "127.0.97.20", port 58229
2026-09-25 10:33:38.287 UTC [11261] LOG: listening on Unix socket "/tmp/.s.PGSQL.58229"
2026-09-25 10:33:38.308 UTC [11264] LOG: database system was shut down at 2026-09-25 10:33:36 UTC
2026-09-25 10:33:38.312 UTC [11261] LOG: database system is ready to accept connections
I20260925 10:33:39.287449 388 mini_postgres.cc:96] Postgres bound to 58229
2026-09-25 10:33:39.304 UTC [11272] FATAL: database "slave" does not exist
127.0.97.20:58229 - accepting connections
I20260925 10:33:39.304889 388 mini_ranger.cc:161] Starting Ranger...
I20260925 10:33:39.330101 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 10:33:39.780176 388 mini_ranger.cc:87] Created ranger Postgres database
I20260925 10:33:39.780301 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-admin
2026-09-25 10:33:40,080 [I] DB FLAVOR :POSTGRES
2026-09-25 10:33:40,081 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:33:40,081 [I] Checking connection..
2026-09-25 10:33:40,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 10:33:40,329 [I] Checking connection passed.
2026-09-25 10:33:40,329 [I] --------- Verifying version history table ---------
2026-09-25 10:33:40,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:33:40,565 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:33:40,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:33:40,822 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:33:40,822 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 10:33:40,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 10:33:41.074 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 10:33:41,080 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 10:33:41,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:33:41,325 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 10:33:41,326 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:33:41,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 10:33:41,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 10:33:41,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dxdd.c.gcp-upstream.internal','N') ;"
2026-09-25 10:33:42,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:33:42,285 [I] Table x_portal_user does not exist in database ranger
2026-09-25 10:33:42,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:33:42,508 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 10:33:42,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:33:42,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:33:43,027 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 10:33:43,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 10:33:43.957 UTC [11444] WARNING: there is no transaction in progress
2026-09-25 10:33:43.972 UTC [11444] WARNING: there is no transaction in progress
2026-09-25 10:33:43.987 UTC [11444] WARNING: there is no transaction in progress
2026-09-25 10:33:44.002 UTC [11444] WARNING: there is no transaction in progress
2026-09-25 10:33:44.020 UTC [11444] WARNING: there is no transaction in progress
2026-09-25 10:33:44.040 UTC [11444] WARNING: there is no transaction in progress
2026-09-25 10:33:44.103 UTC [11444] WARNING: there is no transaction in progress
2026-09-25 10:33:44.110 UTC [11444] WARNING: there is no transaction in progress
2026-09-25 10:33:44.123 UTC [11444] WARNING: there is no transaction in progress
2026-09-25 10:33:44.133 UTC [11444] WARNING: there is no transaction in progress
2026-09-25 10:33:44.143 UTC [11444] WARNING: there is no transaction in progress
2026-09-25 10:33:44.156 UTC [11444] WARNING: there is no transaction in progress
2026-09-25 10:33:44.165 UTC [11444] WARNING: there is no transaction in progress
2026-09-25 10:33:44.171 UTC [11444] WARNING: there is no transaction in progress
2026-09-25 10:33:44.178 UTC [11444] WARNING: there is no transaction in progress
2026-09-25 10:33:44.184 UTC [11444] WARNING: there is no transaction in progress
2026-09-25 10:33:45,133 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 10:33:45,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 10:33:45,347 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 10:33:45,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:33:45,571 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 10:33:45,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:33:45,811 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 10:33:45,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:33:46,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:33:46,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dxdd.c.gcp-upstream.internal';"
2026-09-25 10:33:46,544 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 10:33:46,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 10:33:46,786 [I] DB_PATCHES have already been applied
I20260925 10:33:46.792979 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4ieADW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 10:33:46.793067 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260925 10:33:46.807901 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:57173
Sep 25, 2026 10:33:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:33:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:33:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:33:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:33:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:33:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:33:47 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 10:33:47.929193 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.624s user 0.003s sys 0.035s
Sep 25, 2026 10:34:07 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:34:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 10:34:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:34:07 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:34:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:34:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:34:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:34:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:34:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:34:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:34:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:34:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:34:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:34:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:34:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:34:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:34:07 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:34:08 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 10:34:08.577576 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 10:34:08 dist-test-slave-dxdd krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332448, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 10:34:08.728689 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 10:34:08.747887 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 10:34:09.565189 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 10:34:09.565343 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-kms
I20260925 10:34:09.565464 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:58229
2026-09-25 10:34:10,020 [I] DB FLAVOR :POSTGRES
2026-09-25 10:34:10,020 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:34:10,020 [I] Checking connection
2026-09-25 10:34:10,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 10:34:10,268 [I] connection success
2026-09-25 10:34:10,269 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 10:34:10,269 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 10:34:10,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 10:34:10,511 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 10:34:10,511 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:34:10,511 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 10:34:10,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58229/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 10:34:10,819 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 10:34:11.238830 388 mini_ranger_kms.cc:310] Created kms service
I20260925 10:34:11.448833 388 mini_ranger_kms.cc:326] Created kudu user
I20260925 10:34:11.489116 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 10:34:11 dist-test-slave-dxdd krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332448, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260925 10:34:11.852921 388 mini_ranger_kms.cc:384] Added ranger policy
I20260925 10:34:11.853223 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 10:34:11.853288 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260925 10:34:11.873903 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11624
I20260925 10:34:11.874082 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:57511
Sep 25, 2026 10:34:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:34:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:34:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:34:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:34:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:34:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 10:34:14 dist-test-slave-dxdd krb5kdc[11231](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790332454, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:34:14.141482 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.413s user 0.148s sys 0.315s
I20260925 10:34:14.141742 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 10:34:14.141984 388 mini_ranger_kms.cc:401] 127.0.97.20:57511/kms/v1/keys
Sep 25 10:34:14 dist-test-slave-dxdd krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332454, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 25 10:34:14 dist-test-slave-dxdd krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332454, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 25 10:34:17 dist-test-slave-dxdd krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:34:17.126845 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42799
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42799
--ranger_config_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57511/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:34:17.455982 11675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:34:17.456328 11675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:34:17.456485 11675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:34:17.463996 11675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:34:17.464133 11675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:34:17.464236 11675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:34:17.464340 11675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:34:17.464413 11675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:34:17.474632 11675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:57511/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42799
--ranger_config_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:34:17.475867 11675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:34:17.477106 11675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:34:17.488817 11680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:34:17.488854 11681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:34:17.488888 11683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:34:17.489812 11675 server_base.cc:1034] running on GCE node
I20260925 10:34:17.490671 11675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:34:17.492020 11675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:34:17.493263 11675 hybrid_clock.cc:648] HybridClock initialized: now 1790332457493225 us; error 45 us; skew 500 ppm
I20260925 10:34:17.493599 11675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:34:17 dist-test-slave-dxdd krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:34:17.496968 11675 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:34:17.498375 11675 webserver.cc:460] Webserver started at http://127.0.97.62:36313/ using document root <none> and password file <none>
I20260925 10:34:17.498994 11675 fs_manager.cc:359] Metadata directory not provided
I20260925 10:34:17.499143 11675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:34:17.499445 11675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:34:17 dist-test-slave-dxdd krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260925 10:34:17.557775 11675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "41d1bb2da5d14a4b9d9f7bb52c0f1e5a"
format_stamp: "Formatted at 2026-09-25 10:34:17 on dist-test-slave-dxdd"
server_key: "9e1277489fb5c98ab50f0d62261aa245"
server_key_iv: "d4cc2db0575046695de6b73e42caf496"
server_key_version: "kuduclusterkey@0"
I20260925 10:34:17.558673 11675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "41d1bb2da5d14a4b9d9f7bb52c0f1e5a"
format_stamp: "Formatted at 2026-09-25 10:34:17 on dist-test-slave-dxdd"
server_key: "9e1277489fb5c98ab50f0d62261aa245"
server_key_iv: "d4cc2db0575046695de6b73e42caf496"
server_key_version: "kuduclusterkey@0"
I20260925 10:34:17.564142 11675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:34:17.581128 11690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:34:17.582914 11675 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 10:34:17.583107 11675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "41d1bb2da5d14a4b9d9f7bb52c0f1e5a"
format_stamp: "Formatted at 2026-09-25 10:34:17 on dist-test-slave-dxdd"
server_key: "9e1277489fb5c98ab50f0d62261aa245"
server_key_iv: "d4cc2db0575046695de6b73e42caf496"
server_key_version: "kuduclusterkey@0"
I20260925 10:34:17.583356 11675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:34:17.603176 11675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:34:17.607795 11675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:34:17.608090 11675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:34:17.628796 11675 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42799
I20260925 10:34:17.628798 11741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42799 every 8 connection(s)
I20260925 10:34:17.630460 11675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:34:17.634116 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 11675
I20260925 10:34:17.634231 11742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:34:17.634334 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:34:17.635139 388 external_mini_cluster.cc:1428] Setting key b4385d62b59fe3a09f2527480c30886f
I20260925 10:34:17.648197 11742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a: Bootstrap starting.
I20260925 10:34:17.652048 11742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a: Neither blocks nor log segments found. Creating new log.
Sep 25 10:34:17 dist-test-slave-dxdd krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:34:17.653079 11742 log.cc:824] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a: Log is configured to *not* fsync() on all Append() calls
I20260925 10:34:17.657181 11742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a: No bootstrap required, opened a new log
I20260925 10:34:17.662710 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:17.637266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.0.97.62:42799)
0925 10:34:17.638405 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0925 10:34:17.638421 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:34:17.638479 (+ 58us) server_negotiation.cc:407] Connection header received
0925 10:34:17.641284 (+ 2805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:34:17.641314 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:34:17.641822 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:34:17.642247 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:34:17.643771 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:17.645768 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:17.647562 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:17.648564 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:17.648735 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:17.653907 (+ 5172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:34:17.653918 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:34:17.653938 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:34:17.653986 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:34:17.657936 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:34:17.658846 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:34:17.658853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:34:17.658861 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:34:17.659054 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:34:17.659490 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:34:17.659495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:34:17.659499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:34:17.660192 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:34:17.660521 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 10:34:17.660989 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 10:34:17.661398 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":400,"threads_started":1}
I20260925 10:34:17.666905 11742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d1bb2da5d14a4b9d9f7bb52c0f1e5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42799 } }
I20260925 10:34:17.667248 11742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:34:17.667361 11742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d1bb2da5d14a4b9d9f7bb52c0f1e5a, State: Initialized, Role: FOLLOWER
I20260925 10:34:17.668033 11742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d1bb2da5d14a4b9d9f7bb52c0f1e5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42799 } }
I20260925 10:34:17.668331 11742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:34:17.668476 11742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:34:17.668649 11742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:34:17.671586 11742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d1bb2da5d14a4b9d9f7bb52c0f1e5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42799 } }
I20260925 10:34:17.672137 11742 leader_election.cc:302] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41d1bb2da5d14a4b9d9f7bb52c0f1e5a; no voters:
I20260925 10:34:17.672775 11742 leader_election.cc:288] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:34:17.672887 11747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:34:17.674574 11747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [term 1 LEADER]: Becoming Leader. State: Replica: 41d1bb2da5d14a4b9d9f7bb52c0f1e5a, State: Running, Role: LEADER
I20260925 10:34:17.675184 11742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:34:17.675184 11747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d1bb2da5d14a4b9d9f7bb52c0f1e5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42799 } }
I20260925 10:34:17.683319 11748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41d1bb2da5d14a4b9d9f7bb52c0f1e5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d1bb2da5d14a4b9d9f7bb52c0f1e5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42799 } } }
I20260925 10:34:17.683784 11748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [sys.catalog]: This master's current role is: LEADER
I20260925 10:34:17.684345 11749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41d1bb2da5d14a4b9d9f7bb52c0f1e5a. Latest consensus state: current_term: 1 leader_uuid: "41d1bb2da5d14a4b9d9f7bb52c0f1e5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d1bb2da5d14a4b9d9f7bb52c0f1e5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42799 } } }
I20260925 10:34:17.684607 11749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a [sys.catalog]: This master's current role is: LEADER
I20260925 10:34:17.684687 11754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:34:17.696350 11754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:34:17.718688 11754 catalog_manager.cc:1347] Generated new cluster ID: 190ae1eb1da5410493f4b1735ab58a69
I20260925 10:34:17.718808 11754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:34:17.732577 11754 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:34:17.734496 11754 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:34:17.746345 11742 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 10:34:17.747766 11754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a: Generated new TSK 0
I20260925 10:34:17.748809 11754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:34:18.568135 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57511/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42799
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:34:18 dist-test-slave-dxdd krb5kdc[11231](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790332458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 10:34:19.010926 11788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:34:19.011294 11788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:34:19.011520 11788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:34:19.023479 11788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:34:19.023610 11788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:34:19.023869 11788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:34:19.039661 11788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:57511/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:34:19.041256 11788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:34:19.042858 11788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:34:19.084936 11798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:34:19.086905 11799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:34:19.095526 11802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:34:19.096125 11788 server_base.cc:1034] running on GCE node
I20260925 10:34:19.096832 11788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:34:19.097915 11788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:34:19.099267 11788 hybrid_clock.cc:648] HybridClock initialized: now 1790332459099184 us; error 69 us; skew 500 ppm
I20260925 10:34:19.099604 11788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:34:19 dist-test-slave-dxdd krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:34:19.102891 11788 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:34:19.114063 11788 webserver.cc:460] Webserver started at http://127.0.97.1:45243/ using document root <none> and password file <none>
I20260925 10:34:19.114924 11788 fs_manager.cc:359] Metadata directory not provided
I20260925 10:34:19.115093 11788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:34:19.115456 11788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:34:19 dist-test-slave-dxdd krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260925 10:34:19.151196 11788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "e8845058444f4ff3a81a6fe827defd45"
format_stamp: "Formatted at 2026-09-25 10:34:19 on dist-test-slave-dxdd"
server_key: "c3cd04667e45a04bb1e9096f0f1e11b5"
server_key_iv: "fc9cf3bdc2c311f913e087b4cdbcabc4"
server_key_version: "kuduclusterkey@0"
I20260925 10:34:19.152307 11788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e8845058444f4ff3a81a6fe827defd45"
format_stamp: "Formatted at 2026-09-25 10:34:19 on dist-test-slave-dxdd"
server_key: "c3cd04667e45a04bb1e9096f0f1e11b5"
server_key_iv: "fc9cf3bdc2c311f913e087b4cdbcabc4"
server_key_version: "kuduclusterkey@0"
I20260925 10:34:19.162055 11788 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s
I20260925 10:34:19.207732 11808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:34:19.209926 11788 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.006s sys 0.019s
I20260925 10:34:19.210199 11788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "e8845058444f4ff3a81a6fe827defd45"
format_stamp: "Formatted at 2026-09-25 10:34:19 on dist-test-slave-dxdd"
server_key: "c3cd04667e45a04bb1e9096f0f1e11b5"
server_key_iv: "fc9cf3bdc2c311f913e087b4cdbcabc4"
server_key_version: "kuduclusterkey@0"
I20260925 10:34:19.210605 11788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:34:19.265799 11788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:34:19.271445 11788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:34:19.271862 11788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:34:19.295305 11788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:34:19.315745 11788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:34:19.315976 11788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.001s sys 0.000s
I20260925 10:34:19.316203 11788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:34:19.316327 11788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:34:19 dist-test-slave-dxdd krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260925 10:34:19.807456 11788 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44953
I20260925 10:34:19.807511 11920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44953 every 8 connection(s)
I20260925 10:34:19.809127 11788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:34:19.816323 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 11788
I20260925 10:34:19.816524 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:34:19.817332 388 external_mini_cluster.cc:1428] Setting key e9e72e4c546f8a619bc3234525343b9f
Sep 25 10:34:19 dist-test-slave-dxdd krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:34:19.850908 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:19.812546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42799 (local address 127.0.97.1:57949)
0925 10:34:19.826962 (+ 14416us) negotiation.cc:107] Waiting for socket to connect
0925 10:34:19.827024 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 10:34:19.829534 (+ 2510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:34:19.831330 (+ 1796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:34:19.831342 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:34:19.832096 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:34:19.832982 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:34:19.833009 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:34:19.834560 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:34:19.834566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:34:19.836616 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:34:19.836632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:34:19.837909 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:34:19.837916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:34:19.838182 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:19.839061 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:34:19.839095 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:34:19.841833 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:34:19.846032 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:34:19.846041 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:34:19.846059 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:34:19.846743 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:34:19.847198 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:34:19.847203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:34:19.847206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:34:19.847451 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:34:19.848184 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:34:19.848192 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:34:19.848610 (+ 418us) client_negotiation.cc:769] Sending connection context
0925 10:34:19.848941 (+ 331us) client_negotiation.cc:240] Negotiation successful
0925 10:34:19.849468 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14044,"thread_start_us":17334,"threads_started":1}
I20260925 10:34:19.852068 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:19.811949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57949 (local address 127.0.97.62:42799)
0925 10:34:19.830723 (+ 18774us) server_negotiation.cc:206] Beginning negotiation
0925 10:34:19.830731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:34:19.830752 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:34:19.830802 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:34:19.830806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:34:19.830906 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:34:19.831027 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:34:19.833199 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:19.834353 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:19.836810 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:19.837458 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:19.837599 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:19.842102 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:34:19.842108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:34:19.842113 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:34:19.842161 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:34:19.845848 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:34:19.846889 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:34:19.846895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:34:19.846899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:34:19.847055 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:34:19.847611 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:34:19.847616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:34:19.847620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:34:19.847975 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:34:19.848103 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 10:34:19.851452 (+ 3349us) server_negotiation.cc:299] Negotiation successful
0925 10:34:19.851700 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18649,"thread_start_us":18523,"threads_started":1}
I20260925 10:34:19.854411 11921 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42799
I20260925 10:34:19.854754 11921 heartbeater.cc:464] Registering TS with master...
I20260925 10:34:19.855620 11921 heartbeater.cc:511] Master 127.0.97.62:42799 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260925 10:34:19.858454 11707 ts_manager.cc:194] Registered new tserver with Master: e8845058444f4ff3a81a6fe827defd45 (127.0.97.1:44953)
I20260925 10:34:19.860646 11707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Sep 25 10:34:19 dist-test-slave-dxdd krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:34:19.898197 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57511/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42799
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:34:20.324656 11928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:34:20.324955 11928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:34:20.325117 11928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:34:20.332595 11928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:34:20.332755 11928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:34:20.332924 11928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:34:20.342702 11928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:57511/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:34:20.344020 11928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:34:20.345394 11928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:34:20.360644 11941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:34:20.360987 11942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:34:20.361656 11944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:34:20.361965 11928 server_base.cc:1034] running on GCE node
I20260925 10:34:20.362689 11928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:34:20.363840 11928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:34:20.365090 11928 hybrid_clock.cc:648] HybridClock initialized: now 1790332460365021 us; error 54 us; skew 500 ppm
I20260925 10:34:20.365440 11928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:34:20 dist-test-slave-dxdd krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:34:20.369223 11928 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:34:20.370946 11928 webserver.cc:460] Webserver started at http://127.0.97.2:39599/ using document root <none> and password file <none>
I20260925 10:34:20.371754 11928 fs_manager.cc:359] Metadata directory not provided
I20260925 10:34:20.371879 11928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:34:20.372203 11928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:34:20 dist-test-slave-dxdd krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260925 10:34:20.389740 11928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "d13e8fb42af04669901fcfc12003e9b1"
format_stamp: "Formatted at 2026-09-25 10:34:20 on dist-test-slave-dxdd"
server_key: "866fc82fea9079f7fd432cd8b5d556bd"
server_key_iv: "13b318c5bc7be58bbd12ab9a85d3ae7c"
server_key_version: "kuduclusterkey@0"
I20260925 10:34:20.390695 11928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d13e8fb42af04669901fcfc12003e9b1"
format_stamp: "Formatted at 2026-09-25 10:34:20 on dist-test-slave-dxdd"
server_key: "866fc82fea9079f7fd432cd8b5d556bd"
server_key_iv: "13b318c5bc7be58bbd12ab9a85d3ae7c"
server_key_version: "kuduclusterkey@0"
I20260925 10:34:20.396986 11928 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 10:34:20.413492 11951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:34:20.415079 11928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:34:20.415266 11928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "d13e8fb42af04669901fcfc12003e9b1"
format_stamp: "Formatted at 2026-09-25 10:34:20 on dist-test-slave-dxdd"
server_key: "866fc82fea9079f7fd432cd8b5d556bd"
server_key_iv: "13b318c5bc7be58bbd12ab9a85d3ae7c"
server_key_version: "kuduclusterkey@0"
I20260925 10:34:20.415563 11928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:34:20.435753 11928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:34:20.440267 11928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:34:20.440565 11928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:34:20.441584 11928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:34:20.443063 11928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:34:20.443159 11928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:34:20.443300 11928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:34:20.443372 11928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:34:20.482055 11928 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40231
I20260925 10:34:20.482108 12063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40231 every 8 connection(s)
I20260925 10:34:20.483317 11928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:34:20.487347 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 11928
I20260925 10:34:20.487505 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:34:20.487986 388 external_mini_cluster.cc:1428] Setting key ac45e205c0ba53ddd76906f29fff7c97
Sep 25 10:34:20 dist-test-slave-dxdd krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:34:20.505788 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:20.486224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43739 (local address 127.0.97.62:42799)
0925 10:34:20.486722 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 10:34:20.486728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:34:20.487496 (+ 768us) server_negotiation.cc:407] Connection header received
0925 10:34:20.489747 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:34:20.489753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:34:20.489819 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:34:20.489929 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:34:20.491929 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:20.493053 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:20.495684 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:20.496275 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:20.496399 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:20.499979 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:34:20.499985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:34:20.499988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:34:20.500016 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:34:20.502717 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:34:20.503472 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:34:20.503476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:34:20.503484 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:34:20.503592 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:34:20.504046 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:34:20.504058 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:34:20.504060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:34:20.504330 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:34:20.504492 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 10:34:20.505185 (+ 693us) server_negotiation.cc:299] Negotiation successful
0925 10:34:20.505407 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":238,"threads_started":1}
I20260925 10:34:20.506937 12067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:20.486491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42799 (local address 127.0.97.2:43739)
0925 10:34:20.487340 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 10:34:20.487394 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 10:34:20.489477 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:34:20.490249 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:34:20.490260 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:34:20.490871 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:34:20.491747 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:34:20.491773 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:34:20.493219 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:34:20.493227 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:34:20.495504 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:34:20.495521 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:34:20.496512 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:34:20.496521 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:34:20.496833 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:20.497655 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:34:20.497676 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:34:20.499757 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:34:20.502860 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:34:20.502868 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:34:20.502885 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:34:20.503328 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:34:20.503702 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:34:20.503707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:34:20.503711 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:34:20.503920 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:34:20.504457 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:34:20.504466 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:34:20.504806 (+ 340us) client_negotiation.cc:769] Sending connection context
0925 10:34:20.505077 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 10:34:20.505588 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":397,"threads_started":1}
I20260925 10:34:20.508443 12064 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42799
I20260925 10:34:20.508757 12064 heartbeater.cc:464] Registering TS with master...
I20260925 10:34:20.509557 12064 heartbeater.cc:511] Master 127.0.97.62:42799 requested a full tablet report, sending...
I20260925 10:34:20.510993 11707 ts_manager.cc:194] Registered new tserver with Master: d13e8fb42af04669901fcfc12003e9b1 (127.0.97.2:40231)
I20260925 10:34:20.511759 11707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43739
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:34:20.543020 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57511/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42799
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:34:20.864725 11921 heartbeater.cc:503] Master 127.0.97.62:42799 was elected leader, sending a full tablet report...
W20260925 10:34:20.875263 12070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:34:20.875576 12070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:34:20.875685 12070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:34:20.883417 12070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:34:20.883535 12070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:34:20.883668 12070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:34:20.893664 12070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:57511/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:34:20.894912 12070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:34:20.896131 12070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:34:20.909310 12075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:34:20.909221 12078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:34:20.909265 12076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:34:20.910063 12070 server_base.cc:1034] running on GCE node
I20260925 10:34:20.910702 12070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:34:20.911623 12070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:34:20.912859 12070 hybrid_clock.cc:648] HybridClock initialized: now 1790332460912794 us; error 53 us; skew 500 ppm
I20260925 10:34:20.913153 12070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:34:20 dist-test-slave-dxdd krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:34:20.916622 12070 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:34:20.918040 12070 webserver.cc:460] Webserver started at http://127.0.97.3:34985/ using document root <none> and password file <none>
I20260925 10:34:20.918737 12070 fs_manager.cc:359] Metadata directory not provided
I20260925 10:34:20.918864 12070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:34:20.919173 12070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:34:20 dist-test-slave-dxdd krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260925 10:34:20.933961 12070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "6cf903dce2ca49969eb5960aa424a63d"
format_stamp: "Formatted at 2026-09-25 10:34:20 on dist-test-slave-dxdd"
server_key: "dedde5561a136a0946caae4b91feaab6"
server_key_iv: "95b55f85842fa001dfacf00bcda63b68"
server_key_version: "kuduclusterkey@0"
I20260925 10:34:20.934970 12070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6cf903dce2ca49969eb5960aa424a63d"
format_stamp: "Formatted at 2026-09-25 10:34:20 on dist-test-slave-dxdd"
server_key: "dedde5561a136a0946caae4b91feaab6"
server_key_iv: "95b55f85842fa001dfacf00bcda63b68"
server_key_version: "kuduclusterkey@0"
I20260925 10:34:20.941015 12070 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 10:34:20.956576 12085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:34:20.958057 12070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:34:20.958241 12070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "6cf903dce2ca49969eb5960aa424a63d"
format_stamp: "Formatted at 2026-09-25 10:34:20 on dist-test-slave-dxdd"
server_key: "dedde5561a136a0946caae4b91feaab6"
server_key_iv: "95b55f85842fa001dfacf00bcda63b68"
server_key_version: "kuduclusterkey@0"
I20260925 10:34:20.958492 12070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:34:20.979249 12070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:34:20.983613 12070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:34:20.983896 12070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:34:20.985376 12070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:34:20.986732 12070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:34:20.986801 12070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:34:20.986955 12070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:34:20.987001 12070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:34:21.021294 12070 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42157
I20260925 10:34:21.021339 12197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42157 every 8 connection(s)
I20260925 10:34:21.022588 12070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:34:21.025641 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 12070
I20260925 10:34:21.025805 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:34:21.026435 388 external_mini_cluster.cc:1428] Setting key f4f7cf7c303940236ce08461bbd4809c
Sep 25 10:34:21 dist-test-slave-dxdd krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:34:21.042354 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.025002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38433 (local address 127.0.97.62:42799)
0925 10:34:21.025505 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0925 10:34:21.025513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:34:21.026604 (+ 1091us) server_negotiation.cc:407] Connection header received
0925 10:34:21.028801 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:34:21.028804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:34:21.028933 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:34:21.029047 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:34:21.030654 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.031338 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.032774 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.033249 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.033336 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.036465 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:34:21.036470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:34:21.036473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:34:21.036500 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:34:21.039116 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:34:21.039763 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:34:21.039766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:34:21.039769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:34:21.039878 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:34:21.040385 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:34:21.040389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:34:21.040392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:34:21.040709 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:34:21.040859 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 10:34:21.041721 (+ 862us) server_negotiation.cc:299] Negotiation successful
0925 10:34:21.041946 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":223,"threads_started":1}
I20260925 10:34:21.043186 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.025516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42799 (local address 127.0.97.3:38433)
0925 10:34:21.026421 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0925 10:34:21.026482 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 10:34:21.028490 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:34:21.029240 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:34:21.029249 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:34:21.029793 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:34:21.030493 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:34:21.030512 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.031462 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.031465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:34:21.032658 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:34:21.032669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.033417 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.033421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:34:21.033639 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.034229 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:34:21.034255 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:34:21.036288 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:34:21.039236 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:34:21.039242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:34:21.039253 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:34:21.039662 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:34:21.039971 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:34:21.039974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:34:21.039976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:34:21.040228 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:34:21.040852 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:34:21.040880 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:34:21.041270 (+ 390us) client_negotiation.cc:769] Sending connection context
0925 10:34:21.041595 (+ 325us) client_negotiation.cc:240] Negotiation successful
0925 10:34:21.042025 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":389,"threads_started":1}
I20260925 10:34:21.044445 12198 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42799
I20260925 10:34:21.044694 12198 heartbeater.cc:464] Registering TS with master...
I20260925 10:34:21.045456 12198 heartbeater.cc:511] Master 127.0.97.62:42799 requested a full tablet report, sending...
I20260925 10:34:21.046977 11707 ts_manager.cc:194] Registered new tserver with Master: 6cf903dce2ca49969eb5960aa424a63d (127.0.97.3:42157)
I20260925 10:34:21.048230 11707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38433
I20260925 10:34:21.053992 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:34:21.070398 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.059031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.0.97.62:42799)
0925 10:34:21.059181 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 10:34:21.059185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:34:21.059593 (+ 408us) server_negotiation.cc:407] Connection header received
0925 10:34:21.059746 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:34:21.059750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:34:21.059816 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:34:21.059895 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:34:21.061141 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.061841 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.063388 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.063842 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.063936 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.066066 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:34:21.066070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:34:21.066073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:34:21.066099 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:34:21.068396 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:34:21.068935 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:34:21.068938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:34:21.068940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:34:21.069044 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:34:21.069330 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:34:21.069333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:34:21.069335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:34:21.069548 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:34:21.069715 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:34:21.069944 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 10:34:21.070089 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:34:21.074025 11707 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:34:21.081111 11707 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:34:21.119256 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.095181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.0.97.1:44953)
0925 10:34:21.111148 (+ 15967us) server_negotiation.cc:206] Beginning negotiation
0925 10:34:21.111155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:34:21.111187 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:34:21.111265 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:34:21.111270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:34:21.111455 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:34:21.111600 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:34:21.112351 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.113352 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.116286 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.117573 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.117681 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.117911 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 10:34:21.118698 (+ 787us) server_negotiation.cc:299] Negotiation successful
0925 10:34:21.118855 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15821,"thread_start_us":15677,"threads_started":1}
I20260925 10:34:21.120683 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.094878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44953 (local address 127.0.0.1:40162)
0925 10:34:21.107497 (+ 12619us) negotiation.cc:107] Waiting for socket to connect
0925 10:34:21.107541 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:34:21.107779 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:34:21.111671 (+ 3892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:34:21.111681 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:34:21.111719 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:34:21.112168 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:34:21.112184 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.113476 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.113485 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:34:21.116101 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:34:21.116113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.118142 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.118148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:34:21.118448 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.118505 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 10:34:21.118638 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 10:34:21.119585 (+ 947us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12298,"mutex_wait_us":513,"thread_start_us":13130,"threads_started":1}
I20260925 10:34:21.124007 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.096366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42157 (local address 127.0.0.1:43722)
0925 10:34:21.115391 (+ 19025us) negotiation.cc:107] Waiting for socket to connect
0925 10:34:21.115408 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:34:21.115606 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:34:21.116718 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:34:21.116727 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:34:21.116752 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:34:21.117079 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:34:21.117089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.118906 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.118914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:34:21.121958 (+ 3044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:34:21.121971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.123243 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.123250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:34:21.123408 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.123444 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 10:34:21.123527 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 10:34:21.123666 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18926,"thread_start_us":18605,"threads_started":1}
I20260925 10:34:21.124064 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.095113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40231 (local address 127.0.0.1:42204)
0925 10:34:21.115388 (+ 20275us) negotiation.cc:107] Waiting for socket to connect
0925 10:34:21.115408 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:34:21.115533 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:34:21.116204 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:34:21.116208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:34:21.116237 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:34:21.116671 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:34:21.116683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.118236 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.118243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:34:21.121325 (+ 3082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:34:21.121336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.123285 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.123292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:34:21.123456 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.123490 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 10:34:21.123603 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 10:34:21.123722 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20136,"thread_start_us":20117,"threads_started":1}
I20260925 10:34:21.124835 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.097010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43722 (local address 127.0.97.3:42157)
0925 10:34:21.114756 (+ 17746us) server_negotiation.cc:206] Beginning negotiation
0925 10:34:21.114761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:34:21.115500 (+ 739us) server_negotiation.cc:407] Connection header received
0925 10:34:21.115764 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:34:21.115771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:34:21.115989 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:34:21.116363 (+ 374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:34:21.117255 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.118769 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.122109 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.123035 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.123181 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.124441 (+ 1260us) server_negotiation.cc:1035] Waiting for connection context
0925 10:34:21.124491 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 10:34:21.124605 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17621,"thread_start_us":17470,"threads_started":1}
I20260925 10:34:21.124984 11856 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9a06184dd6445998e0d02017adc031 (DEFAULT_TABLE table=test-table [id=8a38abb4400d4d7eb464903bf363ab9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:34:21.125975 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.095372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.0.97.2:40231)
0925 10:34:21.110963 (+ 15591us) server_negotiation.cc:206] Beginning negotiation
0925 10:34:21.110972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:34:21.115681 (+ 4709us) server_negotiation.cc:407] Connection header received
0925 10:34:21.115767 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:34:21.115773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:34:21.116002 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:34:21.116873 (+ 871us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:34:21.117097 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.118108 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.121478 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.122771 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.122901 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.123101 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 10:34:21.124668 (+ 1567us) server_negotiation.cc:299] Negotiation successful
0925 10:34:21.124819 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15408,"thread_start_us":15582,"threads_started":1}
I20260925 10:34:21.127944 12133 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9a06184dd6445998e0d02017adc031 (DEFAULT_TABLE table=test-table [id=8a38abb4400d4d7eb464903bf363ab9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:34:21.128643 11999 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9a06184dd6445998e0d02017adc031 (DEFAULT_TABLE table=test-table [id=8a38abb4400d4d7eb464903bf363ab9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:34:21.129194 11856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9a06184dd6445998e0d02017adc031. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:34:21.129241 12133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9a06184dd6445998e0d02017adc031. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:34:21.129791 11999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9a06184dd6445998e0d02017adc031. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:34:21.139420 12217 tablet_bootstrap.cc:492] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d: Bootstrap starting.
I20260925 10:34:21.142625 12217 tablet_bootstrap.cc:654] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d: Neither blocks nor log segments found. Creating new log.
I20260925 10:34:21.143806 12217 log.cc:824] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:34:21.144098 12218 tablet_bootstrap.cc:492] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45: Bootstrap starting.
I20260925 10:34:21.145515 12219 tablet_bootstrap.cc:492] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1: Bootstrap starting.
I20260925 10:34:21.148298 12218 tablet_bootstrap.cc:654] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45: Neither blocks nor log segments found. Creating new log.
I20260925 10:34:21.148530 12219 tablet_bootstrap.cc:654] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1: Neither blocks nor log segments found. Creating new log.
I20260925 10:34:21.149158 12217 tablet_bootstrap.cc:492] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d: No bootstrap required, opened a new log
I20260925 10:34:21.149407 12217 ts_tablet_manager.cc:1397] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260925 10:34:21.149484 12218 log.cc:824] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45: Log is configured to *not* fsync() on all Append() calls
I20260925 10:34:21.149459 12219 log.cc:824] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:34:21.156994 12219 tablet_bootstrap.cc:492] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1: No bootstrap required, opened a new log
I20260925 10:34:21.157261 12219 ts_tablet_manager.cc:1397] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260925 10:34:21.158921 12218 tablet_bootstrap.cc:492] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45: No bootstrap required, opened a new log
I20260925 10:34:21.159171 12218 ts_tablet_manager.cc:1397] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.010s
I20260925 10:34:21.159614 12217 raft_consensus.cc:348] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8845058444f4ff3a81a6fe827defd45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "d13e8fb42af04669901fcfc12003e9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40231 } } peers { permanent_uuid: "6cf903dce2ca49969eb5960aa424a63d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } }
I20260925 10:34:21.160036 12217 raft_consensus.cc:374] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:34:21.160136 12217 raft_consensus.cc:729] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf903dce2ca49969eb5960aa424a63d, State: Initialized, Role: FOLLOWER
I20260925 10:34:21.160753 12217 consensus_queue.cc:260] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8845058444f4ff3a81a6fe827defd45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "d13e8fb42af04669901fcfc12003e9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40231 } } peers { permanent_uuid: "6cf903dce2ca49969eb5960aa424a63d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } }
I20260925 10:34:21.165113 12219 raft_consensus.cc:348] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8845058444f4ff3a81a6fe827defd45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "d13e8fb42af04669901fcfc12003e9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40231 } } peers { permanent_uuid: "6cf903dce2ca49969eb5960aa424a63d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } }
I20260925 10:34:21.165457 12219 raft_consensus.cc:374] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:34:21.165560 12219 raft_consensus.cc:729] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13e8fb42af04669901fcfc12003e9b1, State: Initialized, Role: FOLLOWER
I20260925 10:34:21.166133 12219 consensus_queue.cc:260] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8845058444f4ff3a81a6fe827defd45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "d13e8fb42af04669901fcfc12003e9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40231 } } peers { permanent_uuid: "6cf903dce2ca49969eb5960aa424a63d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } }
I20260925 10:34:21.167253 12064 heartbeater.cc:503] Master 127.0.97.62:42799 was elected leader, sending a full tablet report...
I20260925 10:34:21.167713 12219 ts_tablet_manager.cc:1428] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1: Time spent starting tablet: real 0.010s user 0.003s sys 0.007s
I20260925 10:34:21.172119 12218 raft_consensus.cc:348] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8845058444f4ff3a81a6fe827defd45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "d13e8fb42af04669901fcfc12003e9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40231 } } peers { permanent_uuid: "6cf903dce2ca49969eb5960aa424a63d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } }
I20260925 10:34:21.172467 12218 raft_consensus.cc:374] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:34:21.172585 12218 raft_consensus.cc:729] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8845058444f4ff3a81a6fe827defd45, State: Initialized, Role: FOLLOWER
I20260925 10:34:21.173205 12218 consensus_queue.cc:260] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8845058444f4ff3a81a6fe827defd45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "d13e8fb42af04669901fcfc12003e9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40231 } } peers { permanent_uuid: "6cf903dce2ca49969eb5960aa424a63d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } }
I20260925 10:34:21.174523 12198 heartbeater.cc:503] Master 127.0.97.62:42799 was elected leader, sending a full tablet report...
I20260925 10:34:21.175107 12217 ts_tablet_manager.cc:1428] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d: Time spent starting tablet: real 0.026s user 0.022s sys 0.005s
I20260925 10:34:21.175951 12218 ts_tablet_manager.cc:1428] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
W20260925 10:34:21.235384 12065 tablet.cc:2369] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:34:21.274127 12199 tablet.cc:2369] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:34:21.313033 11922 tablet.cc:2369] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:34:21.455257 12224 raft_consensus.cc:482] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:34:21.455581 12224 raft_consensus.cc:504] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8845058444f4ff3a81a6fe827defd45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "d13e8fb42af04669901fcfc12003e9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40231 } } peers { permanent_uuid: "6cf903dce2ca49969eb5960aa424a63d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } }
I20260925 10:34:21.457908 12224 leader_election.cc:288] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8845058444f4ff3a81a6fe827defd45 (127.0.97.1:44953), 6cf903dce2ca49969eb5960aa424a63d (127.0.97.3:42157)
I20260925 10:34:21.464625 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.458371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33907 (local address 127.0.97.1:44953)
0925 10:34:21.458592 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 10:34:21.458602 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:34:21.459276 (+ 674us) server_negotiation.cc:407] Connection header received
0925 10:34:21.459508 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:34:21.459512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:34:21.459568 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:34:21.459680 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:34:21.460110 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.460784 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.462920 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.463711 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.463865 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.463968 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:34:21.464245 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 10:34:21.464369 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:34:21.464740 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.458286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44953 (local address 127.0.97.2:33907)
0925 10:34:21.459162 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0925 10:34:21.459177 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:34:21.459350 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:34:21.459706 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:34:21.459711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:34:21.459741 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:34:21.460001 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:34:21.460008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.460909 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.460914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:34:21.462771 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:34:21.462781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.463925 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.463930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:34:21.464077 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.464106 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 10:34:21.464220 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 10:34:21.464349 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":20,"thread_start_us":466,"threads_started":1}
I20260925 10:34:21.464924 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.458410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60461 (local address 127.0.97.3:42157)
0925 10:34:21.458612 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:34:21.458621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:34:21.459344 (+ 723us) server_negotiation.cc:407] Connection header received
0925 10:34:21.459529 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:34:21.459534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:34:21.459588 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:34:21.459696 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:34:21.460524 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.461508 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.463283 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.464039 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.464189 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.464295 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 10:34:21.464566 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 10:34:21.464688 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 10:34:21.465706 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.458352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42157 (local address 127.0.97.2:60461)
0925 10:34:21.459255 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0925 10:34:21.459269 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:34:21.459384 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:34:21.460122 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:34:21.460125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:34:21.460140 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:34:21.460390 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:34:21.460396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.461645 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.461652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:34:21.463137 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:34:21.463148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.464245 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.464251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:34:21.464416 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.464437 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 10:34:21.464544 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 10:34:21.464661 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":43,"thread_start_us":471,"threads_started":1}
I20260925 10:34:21.466249 11876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9a06184dd6445998e0d02017adc031" candidate_uuid: "d13e8fb42af04669901fcfc12003e9b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8845058444f4ff3a81a6fe827defd45" is_pre_election: true
I20260925 10:34:21.466799 11876 raft_consensus.cc:2454] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13e8fb42af04669901fcfc12003e9b1 in term 0.
I20260925 10:34:21.466759 12153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9a06184dd6445998e0d02017adc031" candidate_uuid: "d13e8fb42af04669901fcfc12003e9b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf903dce2ca49969eb5960aa424a63d" is_pre_election: true
I20260925 10:34:21.467168 12153 raft_consensus.cc:2454] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13e8fb42af04669901fcfc12003e9b1 in term 0.
I20260925 10:34:21.467463 11955 leader_election.cc:302] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d13e8fb42af04669901fcfc12003e9b1, e8845058444f4ff3a81a6fe827defd45; no voters:
I20260925 10:34:21.467898 12224 raft_consensus.cc:2781] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:34:21.468154 12224 raft_consensus.cc:482] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:34:21.468268 12224 raft_consensus.cc:3037] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:34:21.470419 12224 raft_consensus.cc:504] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8845058444f4ff3a81a6fe827defd45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "d13e8fb42af04669901fcfc12003e9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40231 } } peers { permanent_uuid: "6cf903dce2ca49969eb5960aa424a63d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } }
I20260925 10:34:21.471016 12224 leader_election.cc:288] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [CANDIDATE]: Term 1 election: Requested vote from peers e8845058444f4ff3a81a6fe827defd45 (127.0.97.1:44953), 6cf903dce2ca49969eb5960aa424a63d (127.0.97.3:42157)
I20260925 10:34:21.471441 11876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9a06184dd6445998e0d02017adc031" candidate_uuid: "d13e8fb42af04669901fcfc12003e9b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8845058444f4ff3a81a6fe827defd45"
I20260925 10:34:21.471454 12153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9a06184dd6445998e0d02017adc031" candidate_uuid: "d13e8fb42af04669901fcfc12003e9b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf903dce2ca49969eb5960aa424a63d"
I20260925 10:34:21.471642 12153 raft_consensus.cc:3037] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:34:21.471627 11876 raft_consensus.cc:3037] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:34:21.473683 11876 raft_consensus.cc:2454] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e8fb42af04669901fcfc12003e9b1 in term 1.
I20260925 10:34:21.473727 12153 raft_consensus.cc:2454] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e8fb42af04669901fcfc12003e9b1 in term 1.
I20260925 10:34:21.474081 11955 leader_election.cc:302] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d13e8fb42af04669901fcfc12003e9b1, e8845058444f4ff3a81a6fe827defd45; no voters:
I20260925 10:34:21.474393 12224 raft_consensus.cc:2781] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:34:21.474921 12224 raft_consensus.cc:686] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [term 1 LEADER]: Becoming Leader. State: Replica: d13e8fb42af04669901fcfc12003e9b1, State: Running, Role: LEADER
I20260925 10:34:21.475365 12224 consensus_queue.cc:237] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8845058444f4ff3a81a6fe827defd45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "d13e8fb42af04669901fcfc12003e9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40231 } } peers { permanent_uuid: "6cf903dce2ca49969eb5960aa424a63d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } }
I20260925 10:34:21.479842 11706 catalog_manager.cc:5495] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13e8fb42af04669901fcfc12003e9b1 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d13e8fb42af04669901fcfc12003e9b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8845058444f4ff3a81a6fe827defd45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13e8fb42af04669901fcfc12003e9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cf903dce2ca49969eb5960aa424a63d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } health_report { overall_health: UNKNOWN } } }
I20260925 10:34:21.575083 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.569672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.0.97.2:40231)
0925 10:34:21.569882 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 10:34:21.569889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:34:21.569931 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:34:21.570178 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:34:21.570182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:34:21.570273 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:34:21.570480 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:34:21.570842 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.571543 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.572933 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.573334 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.573461 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.573961 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:34:21.574516 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:34:21.574670 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:34:21.574733 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 10:34:21.574816 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:34:21.583829 11876 raft_consensus.cc:1261] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45 [term 1 FOLLOWER]: Refusing update from remote peer d13e8fb42af04669901fcfc12003e9b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:34:21.583830 12153 raft_consensus.cc:1261] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d [term 1 FOLLOWER]: Refusing update from remote peer d13e8fb42af04669901fcfc12003e9b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:34:21.584739 12228 consensus_queue.cc:1035] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8845058444f4ff3a81a6fe827defd45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:34:21.585088 12224 consensus_queue.cc:1035] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cf903dce2ca49969eb5960aa424a63d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:34:21.597177 12231 mvcc.cc:205] Tried to move back new op lower bound from 7333201762635763712 to 7333201762206572544. Current Snapshot: MvccSnapshot[applied={T|T < 7333201762635763712}]
I20260925 10:34:21.605939 12232 mvcc.cc:205] Tried to move back new op lower bound from 7333201762635763712 to 7333201762206572544. Current Snapshot: MvccSnapshot[applied={T|T < 7333201762635763712}]
I20260925 10:34:21.611328 11706 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:34:21.611585 11706 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:34:21.615011 11706 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41d1bb2da5d14a4b9d9f7bb52c0f1e5a: Sending DeleteTablet for 3 replicas of tablet 9d9a06184dd6445998e0d02017adc031
I20260925 10:34:21.616091 11856 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9a06184dd6445998e0d02017adc031 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:34:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40162
I20260925 10:34:21.616155 12133 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9a06184dd6445998e0d02017adc031 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:34:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43722
I20260925 10:34:21.616166 11999 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9a06184dd6445998e0d02017adc031 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:34:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42204
I20260925 10:34:21.620647 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 11788
I20260925 10:34:21.627246 12242 tablet_replica.cc:330] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d: stopping tablet replica
I20260925 10:34:21.627696 12242 raft_consensus.cc:2229] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:34:21.628185 12242 raft_consensus.cc:2258] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:34:21.632700 12242 ts_tablet_manager.cc:1905] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:34:21.638159 12243 tablet_replica.cc:330] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1: stopping tablet replica
I20260925 10:34:21.638718 12243 raft_consensus.cc:2229] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:34:21.638844 12241 tablet_replica.cc:330] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45: stopping tablet replica
I20260925 10:34:21.639242 12241 raft_consensus.cc:2229] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:34:21.639317 12243 raft_consensus.cc:2258] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:34:21.639747 12241 raft_consensus.cc:2258] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:34:21.641206 12242 ts_tablet_manager.cc:1918] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:34:21.641371 12242 log.cc:1196] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/9d9a06184dd6445998e0d02017adc031
I20260925 10:34:21.641815 12242 ts_tablet_manager.cc:1939] T 9d9a06184dd6445998e0d02017adc031 P 6cf903dce2ca49969eb5960aa424a63d: Deleting consensus metadata
I20260925 10:34:21.642508 12241 ts_tablet_manager.cc:1905] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:34:21.643493 11693 catalog_manager.cc:4841] TS 6cf903dce2ca49969eb5960aa424a63d (127.0.97.3:42157): tablet 9d9a06184dd6445998e0d02017adc031 (table test-table [id=8a38abb4400d4d7eb464903bf363ab9c]) successfully deleted
I20260925 10:34:21.643893 12243 ts_tablet_manager.cc:1905] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:34:21.649461 12241 ts_tablet_manager.cc:1918] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:34:21.649616 12241 log.cc:1196] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/9d9a06184dd6445998e0d02017adc031
I20260925 10:34:21.650077 12241 ts_tablet_manager.cc:1939] T 9d9a06184dd6445998e0d02017adc031 P e8845058444f4ff3a81a6fe827defd45: Deleting consensus metadata
I20260925 10:34:21.650972 12243 ts_tablet_manager.cc:1918] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:34:21.651127 12243 log.cc:1196] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/9d9a06184dd6445998e0d02017adc031
I20260925 10:34:21.651654 12243 ts_tablet_manager.cc:1939] T 9d9a06184dd6445998e0d02017adc031 P d13e8fb42af04669901fcfc12003e9b1: Deleting consensus metadata
I20260925 10:34:21.651885 11694 catalog_manager.cc:4841] TS e8845058444f4ff3a81a6fe827defd45 (127.0.97.1:44953): tablet 9d9a06184dd6445998e0d02017adc031 (table test-table [id=8a38abb4400d4d7eb464903bf363ab9c]) successfully deleted
Sep 25 10:34:21 dist-test-slave-dxdd krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:34:21.653107 11691 catalog_manager.cc:4841] TS d13e8fb42af04669901fcfc12003e9b1 (127.0.97.2:40231): tablet 9d9a06184dd6445998e0d02017adc031 (table test-table [id=8a38abb4400d4d7eb464903bf363ab9c]) successfully deleted
I20260925 10:34:21.659838 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.622252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46652 (local address 127.0.97.1:44953)
0925 10:34:21.622493 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 10:34:21.622502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:34:21.634787 (+ 12285us) server_negotiation.cc:407] Connection header received
0925 10:34:21.635112 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:34:21.635122 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:34:21.635269 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:34:21.635360 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:34:21.639725 (+ 4365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.640695 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.647264 (+ 6569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.647941 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.648037 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.653753 (+ 5716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:34:21.653763 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:34:21.653777 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:34:21.653814 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:34:21.656970 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:34:21.657676 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:34:21.657685 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:34:21.657687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:34:21.657810 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:34:21.658189 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:34:21.658194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:34:21.658196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:34:21.658633 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:34:21.658923 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 10:34:21.659267 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 10:34:21.659489 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 10:34:21.660501 11916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:34:21.751093 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 11788
I20260925 10:34:21.769038 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 11928
Sep 25 10:34:21 dist-test-slave-dxdd krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:34:21.785799 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.769807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49490 (local address 127.0.97.2:40231)
0925 10:34:21.769989 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 10:34:21.769993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:34:21.770011 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:34:21.770203 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:34:21.770206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:34:21.770265 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:34:21.770421 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:34:21.772019 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.772847 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.774827 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.775338 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.775473 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.779930 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:34:21.779943 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:34:21.779962 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:34:21.779997 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:34:21.782943 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:34:21.783700 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:34:21.783715 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:34:21.783717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:34:21.783854 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:34:21.784244 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:34:21.784250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:34:21.784253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:34:21.784614 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:34:21.784931 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 10:34:21.785214 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 10:34:21.785444 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:34:21.786471 12059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:34:21.903626 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 11928
I20260925 10:34:21.921131 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 12070
Sep 25 10:34:21 dist-test-slave-dxdd krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:34:21.935070 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:34:21.921951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.0.97.3:42157)
0925 10:34:21.922139 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 10:34:21.922143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:34:21.922161 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:34:21.922360 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:34:21.922364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:34:21.922426 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:34:21.922521 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:34:21.923705 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.924380 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.925643 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:34:21.926023 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:34:21.926111 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:34:21.930096 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:34:21.930110 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:34:21.930126 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:34:21.930155 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:34:21.932702 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:34:21.933433 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:34:21.933440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:34:21.933442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:34:21.933555 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:34:21.933851 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:34:21.933853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:34:21.933855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:34:21.934164 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:34:21.934427 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 10:34:21.934671 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 10:34:21.934816 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:34:21.935724 12193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:34:22.023731 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 12070
I20260925 10:34:22.039650 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 11675
I20260925 10:34:22.040782 11737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:34:22.151999 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 11675
I20260925 10:34:22.172279 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 10:34:22.508495 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 10:34:22.508683 388 mini_ranger.cc:66] Stopping Ranger...
I20260925 10:34:22.854139 388 mini_ranger.cc:68] Stopped Ranger
2026-09-25 10:34:22.854 UTC [11261] LOG: received smart shutdown request
2026-09-25 10:34:22.857 UTC [11261] LOG: background worker "logical replication launcher" (PID 11270) exited with exit code 1
2026-09-25 10:34:22.858 UTC [11265] LOG: shutting down
2026-09-25 10:34:23.013 UTC [11261] LOG: database system is shut down
2026-09-25T10:34:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48725 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:34:23 dist-test-slave-dxdd krb5kdc[12257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:34:23 dist-test-slave-dxdd krb5kdc[12257](info): set up 2 sockets
Sep 25 10:34:23 dist-test-slave-dxdd krb5kdc[12257](info): commencing operation
krb5kdc: starting...
W20260925 10:34:24.226483 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:34:24 dist-test-slave-dxdd krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:34:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:34:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260925 10:34:24.528625 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4ieADW/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/postgres -l logfile start
2026-09-25 10:34:26.885 UTC [12287] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 10:34:26.885 UTC [12287] LOG: listening on IPv4 address "127.0.97.20", port 49657
2026-09-25 10:34:26.888 UTC [12287] LOG: listening on Unix socket "/tmp/.s.PGSQL.49657"
2026-09-25 10:34:26.908 UTC [12290] LOG: database system was shut down at 2026-09-25 10:34:25 UTC
2026-09-25 10:34:26.912 UTC [12287] LOG: database system is ready to accept connections
I20260925 10:34:27.882107 388 mini_postgres.cc:96] Postgres bound to 49657
2026-09-25 10:34:27.899 UTC [12298] FATAL: database "slave" does not exist
127.0.97.20:49657 - accepting connections
I20260925 10:34:27.899670 388 mini_ranger.cc:161] Starting Ranger...
I20260925 10:34:27.929041 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 10:34:28.355044 388 mini_ranger.cc:87] Created ranger Postgres database
I20260925 10:34:28.355135 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-admin
2026-09-25 10:34:28,612 [I] DB FLAVOR :POSTGRES
2026-09-25 10:34:28,612 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:34:28,612 [I] Checking connection..
2026-09-25 10:34:28,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 10:34:28,866 [I] Checking connection passed.
2026-09-25 10:34:28,866 [I] --------- Verifying version history table ---------
2026-09-25 10:34:28,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:34:29,130 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:34:29,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:34:29,379 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:34:29,380 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 10:34:29,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 10:34:29.626 UTC [12362] WARNING: there is no transaction in progress
2026-09-25 10:34:29,632 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 10:34:29,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:34:29,861 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 10:34:29,862 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:34:29,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 10:34:30,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 10:34:30,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dxdd.c.gcp-upstream.internal','N') ;"
2026-09-25 10:34:30,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:34:30,794 [I] Table x_portal_user does not exist in database ranger
2026-09-25 10:34:30,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:34:31,050 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 10:34:31,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:34:31,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:34:31,539 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 10:34:31,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 10:34:32.455 UTC [12470] WARNING: there is no transaction in progress
2026-09-25 10:34:32.471 UTC [12470] WARNING: there is no transaction in progress
2026-09-25 10:34:32.486 UTC [12470] WARNING: there is no transaction in progress
2026-09-25 10:34:32.500 UTC [12470] WARNING: there is no transaction in progress
2026-09-25 10:34:32.518 UTC [12470] WARNING: there is no transaction in progress
2026-09-25 10:34:32.537 UTC [12470] WARNING: there is no transaction in progress
2026-09-25 10:34:32.599 UTC [12470] WARNING: there is no transaction in progress
2026-09-25 10:34:32.605 UTC [12470] WARNING: there is no transaction in progress
2026-09-25 10:34:32.618 UTC [12470] WARNING: there is no transaction in progress
2026-09-25 10:34:32.629 UTC [12470] WARNING: there is no transaction in progress
2026-09-25 10:34:32.638 UTC [12470] WARNING: there is no transaction in progress
2026-09-25 10:34:32.650 UTC [12470] WARNING: there is no transaction in progress
2026-09-25 10:34:32.659 UTC [12470] WARNING: there is no transaction in progress
2026-09-25 10:34:32.664 UTC [12470] WARNING: there is no transaction in progress
2026-09-25 10:34:32.670 UTC [12470] WARNING: there is no transaction in progress
2026-09-25 10:34:32.676 UTC [12470] WARNING: there is no transaction in progress
2026-09-25 10:34:33,264 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 10:34:33,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 10:34:33,515 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 10:34:33,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:34:33,740 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 10:34:33,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:34:33,999 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 10:34:33,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:34:34,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:34:34,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dxdd.c.gcp-upstream.internal';"
2026-09-25 10:34:34,712 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 10:34:34,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 10:34:34,934 [I] DB_PATCHES have already been applied
I20260925 10:34:34.939765 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4ieADW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 10:34:34.939843 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260925 10:34:34.954030 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:44307
Sep 25, 2026 10:34:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:34:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:34:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:34:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:34:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:34:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:34:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 10:34:36.084820 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.185s user 0.005s sys 0.035s
Sep 25, 2026 10:34:56 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:34:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 10:34:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:34:56 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:34:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:34:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:34:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:34:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:34:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:34:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:34:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:34:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:34:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:34:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:34:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:34:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:34:56 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:34:57 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 10:34:57.622187 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 10:34:57 dist-test-slave-dxdd krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332497, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 10:34:57.798970 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 10:34:57.818682 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 10:34:58.641970 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 10:34:58.642089 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-kms
I20260925 10:34:58.642143 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:49657
2026-09-25 10:34:59,041 [I] DB FLAVOR :POSTGRES
2026-09-25 10:34:59,042 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:34:59,042 [I] Checking connection
2026-09-25 10:34:59,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 10:34:59,300 [I] connection success
2026-09-25 10:34:59,300 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 10:34:59,300 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 10:34:59,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 10:34:59,557 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 10:34:59,557 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:34:59,557 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 10:34:59,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49657/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 10:34:59,852 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 10:35:00.327433 388 mini_ranger_kms.cc:310] Created kms service
I20260925 10:35:00.534245 388 mini_ranger_kms.cc:326] Created kudu user
I20260925 10:35:00.580965 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 10:35:00 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332497, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260925 10:35:00.971458 388 mini_ranger_kms.cc:384] Added ranger policy
I20260925 10:35:00.971757 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task4ieADW/build/asan/bin/postgresql.jar:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4ieADW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 10:35:00.971832 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260925 10:35:00.989537 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12650
I20260925 10:35:00.989666 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:33909
Sep 25, 2026 10:35:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:35:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:35:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:35:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:35:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:35:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 10:35:03.262295 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.463s user 0.074s sys 0.320s
I20260925 10:35:03.262552 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 10:35:03.262735 388 mini_ranger_kms.cc:401] 127.0.97.20:33909/kms/v1/keys
Sep 25 10:35:03 dist-test-slave-dxdd krb5kdc[12257](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790332503, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 10:35:03 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332503, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 25 10:35:04 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332503, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 25 10:35:06 dist-test-slave-dxdd krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:35:06.342200 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39895
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39895
--ranger_config_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33909/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:35:06.663465 12701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:06.663738 12701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:06.663889 12701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:06.671305 12701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:35:06.671500 12701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:06.671603 12701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:06.671708 12701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:35:06.671798 12701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:35:06.681857 12701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:33909/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39895
--ranger_config_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:06.683179 12701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:06.684542 12701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:06.697387 12706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:06.697358 12707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:06.698668 12709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:06.698910 12701 server_base.cc:1034] running on GCE node
I20260925 10:35:06.699822 12701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:06.701455 12701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:06.702689 12701 hybrid_clock.cc:648] HybridClock initialized: now 1790332506702626 us; error 52 us; skew 500 ppm
I20260925 10:35:06.703078 12701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:06 dist-test-slave-dxdd krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:06.713527 12701 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:35:06.715127 12701 webserver.cc:460] Webserver started at http://127.0.97.62:41979/ using document root <none> and password file <none>
I20260925 10:35:06.715780 12701 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:06.715875 12701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:06.716216 12701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:35:06 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260925 10:35:06.773609 12701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb"
format_stamp: "Formatted at 2026-09-25 10:35:06 on dist-test-slave-dxdd"
server_key: "dfcb20b2b054eaca7e32d1355d605144"
server_key_iv: "846a28ee7d4872e3406ec5a7d96f836c"
server_key_version: "kuduclusterkey@0"
I20260925 10:35:06.774654 12701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb"
format_stamp: "Formatted at 2026-09-25 10:35:06 on dist-test-slave-dxdd"
server_key: "dfcb20b2b054eaca7e32d1355d605144"
server_key_iv: "846a28ee7d4872e3406ec5a7d96f836c"
server_key_version: "kuduclusterkey@0"
I20260925 10:35:06.781165 12701 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260925 10:35:06.800417 12716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:06.802199 12701 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 10:35:06.802408 12701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb"
format_stamp: "Formatted at 2026-09-25 10:35:06 on dist-test-slave-dxdd"
server_key: "dfcb20b2b054eaca7e32d1355d605144"
server_key_iv: "846a28ee7d4872e3406ec5a7d96f836c"
server_key_version: "kuduclusterkey@0"
I20260925 10:35:06.802655 12701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:06.819134 12701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:06.823536 12701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:06.823849 12701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:06.842823 12701 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39895
I20260925 10:35:06.842831 12767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39895 every 8 connection(s)
I20260925 10:35:06.844388 12701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:35:06.848095 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:06.854642 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 12701
I20260925 10:35:06.854846 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:35:06.855630 388 external_mini_cluster.cc:1428] Setting key f5e10a989a7ec0e05418fb1f774a7b6e
I20260925 10:35:06.858886 12768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb: Bootstrap starting.
I20260925 10:35:06.863806 12768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:06.865453 12768 log.cc:824] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:06.875038 12768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb: No bootstrap required, opened a new log
Sep 25 10:35:06 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:06.885372 12768 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39895 } }
I20260925 10:35:06.885842 12768 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:06.885949 12768 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ec021f327d4aa5ad21ffb97f32ddeb, State: Initialized, Role: FOLLOWER
I20260925 10:35:06.886693 12768 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39895 } }
I20260925 10:35:06.886977 12768 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:35:06.887115 12768 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:35:06.887296 12768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:06.890671 12768 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39895 } }
I20260925 10:35:06.891232 12768 leader_election.cc:302] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3ec021f327d4aa5ad21ffb97f32ddeb; no voters:
I20260925 10:35:06.891911 12768 leader_election.cc:288] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:35:06.892073 12773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:35:06.893767 12771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:06.857612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.0.97.62:39895)
0925 10:35:06.873133 (+ 15521us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:06.873145 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:06.873183 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:35:06.874948 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:06.874970 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:06.875424 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:06.875882 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:06.877267 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:06.878671 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:06.879924 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:06.880624 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:06.880829 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:06.884625 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:06.884633 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:06.884647 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:06.884687 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:06.888164 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:06.888882 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:06.888890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:06.888898 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:06.889064 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:06.889450 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:06.889458 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:06.889463 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:06.890083 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:06.891399 (+ 1316us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:06.891664 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 10:35:06.892130 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15287,"thread_start_us":15069,"threads_started":1}
I20260925 10:35:06.900172 12773 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 1 LEADER]: Becoming Leader. State: Replica: a3ec021f327d4aa5ad21ffb97f32ddeb, State: Running, Role: LEADER
I20260925 10:35:06.900245 12768 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:35:06.900879 12773 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39895 } }
I20260925 10:35:06.908180 12774 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39895 } } }
I20260925 10:35:06.908578 12774 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:06.909693 12775 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3ec021f327d4aa5ad21ffb97f32ddeb. Latest consensus state: current_term: 1 leader_uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39895 } } }
I20260925 10:35:06.910048 12775 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:06.924407 12780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:35:06.946614 12780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:35:06.963321 12780 catalog_manager.cc:1347] Generated new cluster ID: 61de0a380ebb4c028c2f72160e57e292
I20260925 10:35:06.963449 12780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:35:06.964761 12768 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 10:35:07.030740 12780 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:35:07.033213 12780 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:35:07.045399 12780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb: Generated new TSK 0
I20260925 10:35:07.046586 12780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:35:07.798074 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33909/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39895
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:35:08 dist-test-slave-dxdd krb5kdc[12257](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790332508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 10:35:08.180691 12814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:08.181074 12814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:08.181290 12814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:08.192647 12814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:08.192777 12814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:08.193011 12814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:35:08.208510 12814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:33909/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:08.210103 12814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:08.211853 12814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:08.227931 12824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:08.227962 12825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:08.228122 12827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:08.229986 12814 server_base.cc:1034] running on GCE node
I20260925 10:35:08.230719 12814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:08.231913 12814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:08.233197 12814 hybrid_clock.cc:648] HybridClock initialized: now 1790332508233104 us; error 81 us; skew 500 ppm
I20260925 10:35:08.233563 12814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:08 dist-test-slave-dxdd krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:08.237672 12814 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:35:08.239584 12814 webserver.cc:460] Webserver started at http://127.0.97.1:41915/ using document root <none> and password file <none>
I20260925 10:35:08.240449 12814 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:08.240577 12814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:08.240912 12814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:35:08 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260925 10:35:08.267836 12814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "782672c1801c4db1b897b0867a6cd120"
format_stamp: "Formatted at 2026-09-25 10:35:08 on dist-test-slave-dxdd"
server_key: "5265d6de38b773d340d59263d6adaa5a"
server_key_iv: "616e395c87839a828e8b71a287736d40"
server_key_version: "kuduclusterkey@0"
I20260925 10:35:08.268993 12814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "782672c1801c4db1b897b0867a6cd120"
format_stamp: "Formatted at 2026-09-25 10:35:08 on dist-test-slave-dxdd"
server_key: "5265d6de38b773d340d59263d6adaa5a"
server_key_iv: "616e395c87839a828e8b71a287736d40"
server_key_version: "kuduclusterkey@0"
I20260925 10:35:08.287866 12814 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.013s sys 0.004s
I20260925 10:35:08.309957 12834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:08.311908 12814 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.002s
I20260925 10:35:08.312166 12814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "782672c1801c4db1b897b0867a6cd120"
format_stamp: "Formatted at 2026-09-25 10:35:08 on dist-test-slave-dxdd"
server_key: "5265d6de38b773d340d59263d6adaa5a"
server_key_iv: "616e395c87839a828e8b71a287736d40"
server_key_version: "kuduclusterkey@0"
I20260925 10:35:08.312497 12814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:08.364166 12814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:08.370494 12814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:08.370909 12814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:08.378711 12814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:08.381834 12814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:08.381994 12814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 10:35:08.382220 12814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:08.382393 12814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:08.444957 12814 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44351
I20260925 10:35:08.445012 12946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44351 every 8 connection(s)
I20260925 10:35:08.446811 12814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:35:08.450789 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 12814
I20260925 10:35:08.450987 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:35:08.451802 388 external_mini_cluster.cc:1428] Setting key 784ffcf4129d59f96affb849fc878070
Sep 25 10:35:08 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:08.509188 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:08.458540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56789 (local address 127.0.97.62:39895)
0925 10:35:08.478209 (+ 19669us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:08.478218 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:08.478248 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:35:08.478447 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:08.478453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:08.478567 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:08.489193 (+ 10626us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:08.489426 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:08.490995 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:08.494385 (+ 3390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:08.495308 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:08.496596 (+ 1288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:08.500640 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:08.500649 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:08.500657 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:08.500701 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:08.504657 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:08.505628 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:08.505636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:08.505642 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:08.505791 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:08.506387 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:08.506395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:08.506401 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:08.506784 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:08.508413 (+ 1629us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:08.508535 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 10:35:08.508792 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19466,"thread_start_us":19732,"threads_started":1}
I20260925 10:35:08.509876 12949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:08.456029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39895 (local address 127.0.97.1:56789)
0925 10:35:08.465025 (+ 8996us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:08.465097 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:08.469960 (+ 4863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:08.486847 (+ 16887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:08.486868 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:08.487792 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:08.488940 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:08.488972 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:08.491227 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:08.491236 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:08.494086 (+ 2850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:08.494107 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:08.495643 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:08.495653 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:08.496056 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:08.497012 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:08.497055 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:08.500367 (+ 3312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:08.504821 (+ 4454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:08.504834 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:08.504854 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:08.505452 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:08.505920 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:08.505927 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:08.505934 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:08.506202 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:08.506943 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:08.506951 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:08.507401 (+ 450us) client_negotiation.cc:769] Sending connection context
0925 10:35:08.507812 (+ 411us) client_negotiation.cc:240] Negotiation successful
0925 10:35:08.508382 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8705,"thread_start_us":8479,"threads_started":1}
I20260925 10:35:08.512327 12947 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39895
I20260925 10:35:08.512679 12947 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260925 10:35:08.513481 12947 heartbeater.cc:511] Master 127.0.97.62:39895 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260925 10:35:08.516232 12733 ts_manager.cc:194] Registered new tserver with Master: 782672c1801c4db1b897b0867a6cd120 (127.0.97.1:44351)
I20260925 10:35:08.518262 12733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:35:08.552343 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33909/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39895
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:35:08 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 25 10:35:08 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260925 10:35:08.992888 12953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:08.993376 12953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:08.993602 12953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:09.004827 12953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:09.005018 12953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:09.005247 12953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:35:09.020783 12953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:33909/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:09.022718 12953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:09.024421 12953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:09.073443 12960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:09.080817 12961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:09.082010 12964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:09.083165 12953 server_base.cc:1034] running on GCE node
I20260925 10:35:09.084075 12953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:09.090067 12953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:09.091375 12953 hybrid_clock.cc:648] HybridClock initialized: now 1790332509091271 us; error 101 us; skew 500 ppm
I20260925 10:35:09.091804 12953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:09 dist-test-slave-dxdd krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:09.095983 12953 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:35:09.116364 12953 webserver.cc:460] Webserver started at http://127.0.97.2:39981/ using document root <none> and password file <none>
I20260925 10:35:09.117479 12953 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:09.117705 12953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:09.118132 12953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:35:09 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260925 10:35:09.137820 12953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "f01981365d43439a9a3810d94f455550"
format_stamp: "Formatted at 2026-09-25 10:35:09 on dist-test-slave-dxdd"
server_key: "80b6f461dca442ecb3ac54779a97b253"
server_key_iv: "730e757bb0b99b9a0434e2ed9f0d70dd"
server_key_version: "kuduclusterkey@0"
I20260925 10:35:09.138983 12953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f01981365d43439a9a3810d94f455550"
format_stamp: "Formatted at 2026-09-25 10:35:09 on dist-test-slave-dxdd"
server_key: "80b6f461dca442ecb3ac54779a97b253"
server_key_iv: "730e757bb0b99b9a0434e2ed9f0d70dd"
server_key_version: "kuduclusterkey@0"
I20260925 10:35:09.152278 12953 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.006s sys 0.005s
I20260925 10:35:09.200768 12970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:09.202977 12953 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.007s sys 0.014s
I20260925 10:35:09.203255 12953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "f01981365d43439a9a3810d94f455550"
format_stamp: "Formatted at 2026-09-25 10:35:09 on dist-test-slave-dxdd"
server_key: "80b6f461dca442ecb3ac54779a97b253"
server_key_iv: "730e757bb0b99b9a0434e2ed9f0d70dd"
server_key_version: "kuduclusterkey@0"
I20260925 10:35:09.203675 12953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:09.246995 12953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:09.252164 12953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:09.252544 12953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:09.254668 12953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:09.256456 12953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:09.256618 12953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 10:35:09.256803 12953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:09.257010 12953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:09.307613 12953 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45187
I20260925 10:35:09.307633 13089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45187 every 8 connection(s)
I20260925 10:35:09.309072 12953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:35:09.310277 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 12953
I20260925 10:35:09.310503 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:35:09.311178 388 external_mini_cluster.cc:1428] Setting key aa9cde4bf68e68c699867e5db0bd9879
Sep 25 10:35:09 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260925 10:35:09.347779 13092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:09.314615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56793 (local address 127.0.97.62:39895)
0925 10:35:09.329144 (+ 14529us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:09.329152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:09.329173 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:35:09.329234 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:09.329242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:09.329348 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:09.329469 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:09.331507 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:09.332862 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:09.335296 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:09.336123 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:09.336267 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:09.340881 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:09.340887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:09.340892 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:09.340942 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:09.343924 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:09.344849 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:09.344857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:09.344863 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:09.345004 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:09.345542 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:09.345550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:09.345556 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:09.345951 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:09.346104 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:09.346998 (+ 894us) server_negotiation.cc:299] Negotiation successful
0925 10:35:09.347272 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14376,"thread_start_us":14975,"threads_started":1}
I20260925 10:35:09.348778 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:09.314636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39895 (local address 127.0.97.2:56793)
0925 10:35:09.325802 (+ 11166us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:09.325858 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:09.328298 (+ 2440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:09.329775 (+ 1477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:09.329787 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:09.330479 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:09.331285 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:09.331309 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:09.333062 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:09.333068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:09.335120 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:09.335134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:09.336394 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:09.336403 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:09.336673 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:09.337553 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:09.337583 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:09.340550 (+ 2967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:09.344111 (+ 3561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:09.344120 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:09.344138 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:09.344695 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:09.345138 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:09.345142 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:09.345146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:09.345393 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:09.346106 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:09.346114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:09.346549 (+ 435us) client_negotiation.cc:769] Sending connection context
0925 10:35:09.346849 (+ 300us) client_negotiation.cc:240] Negotiation successful
0925 10:35:09.347390 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10837,"thread_start_us":13197,"threads_started":1}
I20260925 10:35:09.350489 13090 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39895
I20260925 10:35:09.350822 13090 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:09.351526 13090 heartbeater.cc:511] Master 127.0.97.62:39895 requested a full tablet report, sending...
I20260925 10:35:09.353118 12733 ts_manager.cc:194] Registered new tserver with Master: f01981365d43439a9a3810d94f455550 (127.0.97.2:45187)
I20260925 10:35:09.354226 12733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:35:09.377564 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33909/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39895
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:35:09.522015 12947 heartbeater.cc:503] Master 127.0.97.62:39895 was elected leader, sending a full tablet report...
W20260925 10:35:09.697467 13096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:09.697762 13096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:09.697937 13096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:09.705755 13096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:09.705892 13096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:09.706060 13096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:35:09.715867 13096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:33909/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:09.717136 13096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:09.718524 13096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:09.731369 13101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:09.731302 13104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:09.731407 13102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:09.732214 13096 server_base.cc:1034] running on GCE node
I20260925 10:35:09.732818 13096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:09.733762 13096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:09.734948 13096 hybrid_clock.cc:648] HybridClock initialized: now 1790332509734902 us; error 45 us; skew 500 ppm
I20260925 10:35:09.735260 13096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:09 dist-test-slave-dxdd krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:09.738211 13096 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:35:09.739583 13096 webserver.cc:460] Webserver started at http://127.0.97.3:41363/ using document root <none> and password file <none>
I20260925 10:35:09.740226 13096 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:09.740360 13096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:09.740656 13096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:35:09 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260925 10:35:09.756742 13096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "98b8c7d50b4843559ad50b51b61acc49"
format_stamp: "Formatted at 2026-09-25 10:35:09 on dist-test-slave-dxdd"
server_key: "a16aeed93f4cc0b4e07d43274fd4bbd7"
server_key_iv: "82606f0dc97acf6271627545b5019c24"
server_key_version: "kuduclusterkey@0"
I20260925 10:35:09.757737 13096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "98b8c7d50b4843559ad50b51b61acc49"
format_stamp: "Formatted at 2026-09-25 10:35:09 on dist-test-slave-dxdd"
server_key: "a16aeed93f4cc0b4e07d43274fd4bbd7"
server_key_iv: "82606f0dc97acf6271627545b5019c24"
server_key_version: "kuduclusterkey@0"
I20260925 10:35:09.764037 13096 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260925 10:35:09.780951 13111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:09.782711 13096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:35:09.782876 13096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "98b8c7d50b4843559ad50b51b61acc49"
format_stamp: "Formatted at 2026-09-25 10:35:09 on dist-test-slave-dxdd"
server_key: "a16aeed93f4cc0b4e07d43274fd4bbd7"
server_key_iv: "82606f0dc97acf6271627545b5019c24"
server_key_version: "kuduclusterkey@0"
I20260925 10:35:09.783123 13096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:09.812453 13096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:09.816562 13096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:09.816869 13096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:09.817899 13096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:09.819350 13096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:09.819428 13096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:09.819571 13096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:09.819635 13096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:09.853579 13096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36127
I20260925 10:35:09.853590 13223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36127 every 8 connection(s)
I20260925 10:35:09.855057 13096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:35:09.861256 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 13096
I20260925 10:35:09.861446 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:35:09.861905 388 external_mini_cluster.cc:1428] Setting key 8b40c4f31566ea9eca57690d65fe91fd
Sep 25 10:35:09 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:09.874571 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:09.857775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35019 (local address 127.0.97.62:39895)
0925 10:35:09.858192 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:09.858197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:09.859210 (+ 1013us) server_negotiation.cc:407] Connection header received
0925 10:35:09.860968 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:09.860974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:09.861057 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:09.861183 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:09.862841 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:09.863876 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:09.865475 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:09.865873 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:09.865967 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:09.869065 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:09.869069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:09.869072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:09.869110 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:09.871511 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:09.872317 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:09.872325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:09.872331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:09.872496 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:09.873086 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:09.873090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:09.873093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:09.873360 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:09.873496 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:09.874113 (+ 617us) server_negotiation.cc:299] Negotiation successful
0925 10:35:09.874266 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":189,"threads_started":1}
I20260925 10:35:09.875690 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:09.858363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39895 (local address 127.0.97.3:35019)
0925 10:35:09.859052 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:09.859092 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:09.860650 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:09.861436 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:09.861444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:09.861881 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:09.862629 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:09.862654 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:09.864045 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:09.864054 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:09.865348 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:09.865359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:09.866058 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:09.866062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:09.866247 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:09.866963 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:09.866988 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:09.868882 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:09.871627 (+ 2745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:09.871633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:09.871644 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:09.872157 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:09.872617 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:09.872623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:09.872627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:09.872948 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:09.873500 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:09.873506 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:09.873782 (+ 276us) client_negotiation.cc:769] Sending connection context
0925 10:35:09.873991 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 10:35:09.874370 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":301,"threads_started":1}
I20260925 10:35:09.877028 13224 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39895
I20260925 10:35:09.877298 13224 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:09.877936 13224 heartbeater.cc:511] Master 127.0.97.62:39895 requested a full tablet report, sending...
I20260925 10:35:09.879284 12733 ts_manager.cc:194] Registered new tserver with Master: 98b8c7d50b4843559ad50b51b61acc49 (127.0.97.3:36127)
I20260925 10:35:09.880015 12733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35019
I20260925 10:35:09.889140 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:35:09.890868 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 12814
Sep 25 10:35:09 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:35:09.906697 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:09.891773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.0.97.1:44351)
0925 10:35:09.892381 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:09.892388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:09.892416 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:35:09.892469 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:09.892475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:09.892646 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:09.892766 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:09.894049 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:09.895116 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:09.896518 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:09.897020 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:09.897112 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:09.901362 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:09.901373 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:09.901385 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:09.901429 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:09.904295 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:09.904924 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:09.904930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:09.904933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:09.905071 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:09.905383 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:09.905386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:09.905388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:09.905682 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:09.905899 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:09.906166 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 10:35:09.906423 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":316,"threads_started":1}
I20260925 10:35:09.908015 12942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:09.990245 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 12814
I20260925 10:35:10.005995 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 12953
Sep 25 10:35:10 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:35:10.021677 13231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:10.006978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.0.97.2:45187)
0925 10:35:10.007573 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:10.007581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:10.007607 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:35:10.007665 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:10.007671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:10.007920 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:10.008044 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:10.009396 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:10.010582 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:10.011893 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:10.012413 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:10.012507 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:10.016736 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:10.016747 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:10.016761 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:10.016791 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:10.019375 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:10.019981 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:10.019987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:10.019989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:10.020106 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:10.020405 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:10.020408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:10.020410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:10.020700 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:10.020933 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:10.021107 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 10:35:10.021285 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":353,"threads_started":1}
I20260925 10:35:10.022970 13085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:10.105212 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 12953
I20260925 10:35:10.120666 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 13096
Sep 25 10:35:10 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:35:10.136574 13233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:10.121685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47734 (local address 127.0.97.3:36127)
0925 10:35:10.122416 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:10.122423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:10.122451 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:35:10.122507 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:10.122514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:10.122727 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:10.122865 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:10.124117 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:10.125094 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:10.126271 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:10.126977 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:10.127073 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:10.131236 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:10.131247 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:10.131260 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:10.131291 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:10.134012 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:10.134735 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:10.134742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:10.134744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:10.134878 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:10.135234 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:10.135237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:10.135239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:10.135554 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:10.135771 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:10.135995 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 10:35:10.136177 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":380,"threads_started":1}
I20260925 10:35:10.137862 13219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:10.219508 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 13096
I20260925 10:35:10.235159 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 12701
I20260925 10:35:10.236377 12763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:10.341235 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 12701
I20260925 10:35:10.365257 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39895
--webserver_interface=127.0.97.62
--webserver_port=41979
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39895
--ranger_config_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33909/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33909/kms with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:35:10.675657 13236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:10.675923 13236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:10.676139 13236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:10.683101 13236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:35:10.683190 13236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:10.683285 13236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:10.683395 13236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:35:10.683488 13236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:35:10.693054 13236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33909/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39895
--ranger_config_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=41979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:10.694355 13236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:10.695639 13236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:10.707726 13241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:10.707701 13242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:10.707790 13244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:10.708662 13236 server_base.cc:1034] running on GCE node
I20260925 10:35:10.709362 13236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:10.710824 13236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:10.712059 13236 hybrid_clock.cc:648] HybridClock initialized: now 1790332510712020 us; error 60 us; skew 500 ppm
I20260925 10:35:10.712400 13236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:10 dist-test-slave-dxdd krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:10.715341 13236 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:35:10.716780 13236 webserver.cc:460] Webserver started at http://127.0.97.62:41979/ using document root <none> and password file <none>
I20260925 10:35:10.717413 13236 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:10.717564 13236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:35:10 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260925 10:35:10.807301 13236 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 10:35:10.810421 13252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:10.811843 13236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:35:10.812040 13236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb"
format_stamp: "Formatted at 2026-09-25 10:35:06 on dist-test-slave-dxdd"
server_key: "dfcb20b2b054eaca7e32d1355d605144"
server_key_iv: "846a28ee7d4872e3406ec5a7d96f836c"
server_key_version: "kuduclusterkey@0"
I20260925 10:35:10.812996 13236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:10.847235 13236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:10.850795 13236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:10.851094 13236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:10.869088 13236 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39895
I20260925 10:35:10.869112 13303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39895 every 8 connection(s)
I20260925 10:35:10.870493 13236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:35:10.877113 13304 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:35:10.879109 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 13236
I20260925 10:35:10.880075 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33909/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44351
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39895
--webserver_port=41915
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33909/kms with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:35:10.881157 13304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb: Bootstrap starting.
I20260925 10:35:10.901114 13304 log.cc:824] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:10.909508 13304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:35:10.910053 13304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb: Bootstrap complete.
I20260925 10:35:10.919041 13304 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39895 } }
I20260925 10:35:10.919929 13304 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ec021f327d4aa5ad21ffb97f32ddeb, State: Initialized, Role: FOLLOWER
I20260925 10:35:10.920513 13304 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39895 } }
I20260925 10:35:10.920750 13304 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:35:10.920887 13304 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:35:10.921064 13304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:35:10.924652 13304 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39895 } }
I20260925 10:35:10.925128 13304 leader_election.cc:302] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3ec021f327d4aa5ad21ffb97f32ddeb; no voters:
I20260925 10:35:10.925736 13304 leader_election.cc:288] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:35:10.925889 13308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:35:10.927104 13308 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [term 2 LEADER]: Becoming Leader. State: Replica: a3ec021f327d4aa5ad21ffb97f32ddeb, State: Running, Role: LEADER
I20260925 10:35:10.927687 13308 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39895 } }
I20260925 10:35:10.927959 13304 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:35:10.930217 13309 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39895 } } }
I20260925 10:35:10.930667 13309 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:10.932487 13313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:35:10.932322 13310 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3ec021f327d4aa5ad21ffb97f32ddeb. Latest consensus state: current_term: 2 leader_uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ec021f327d4aa5ad21ffb97f32ddeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39895 } } }
I20260925 10:35:10.932790 13310 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:10.938097 13313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:35:10.939457 13313 catalog_manager.cc:1259] Loaded cluster ID: 61de0a380ebb4c028c2f72160e57e292
I20260925 10:35:10.939596 13313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:35:10.943270 13313 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:35:10.944553 13313 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb: Loaded TSK: 0
I20260925 10:35:10.945945 13313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:35:10.955418 13304 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260925 10:35:11.299849 13306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:11.300415 13306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:11.300611 13306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:11.312706 13306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:11.312949 13306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:11.313194 13306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:35:11.329603 13306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33909/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41915
--tserver_master_addrs=127.0.97.62:39895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:11.331306 13306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:11.333027 13306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:11.367341 13341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:11.370731 13342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:11.379733 13344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:11.380699 13306 server_base.cc:1034] running on GCE node
I20260925 10:35:11.381510 13306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:11.415879 13306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:11.417265 13306 hybrid_clock.cc:648] HybridClock initialized: now 1790332511417121 us; error 150 us; skew 500 ppm
I20260925 10:35:11.417793 13306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:11 dist-test-slave-dxdd krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:11.421506 13306 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:35:11.433434 13306 webserver.cc:460] Webserver started at http://127.0.97.1:41915/ using document root <none> and password file <none>
I20260925 10:35:11.434634 13306 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:11.434870 13306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:35:11 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260925 10:35:11.486799 13306 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.006s sys 0.001s
I20260925 10:35:11.503590 13352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:11.505584 13306 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.003s sys 0.006s
I20260925 10:35:11.505856 13306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "782672c1801c4db1b897b0867a6cd120"
format_stamp: "Formatted at 2026-09-25 10:35:08 on dist-test-slave-dxdd"
server_key: "5265d6de38b773d340d59263d6adaa5a"
server_key_iv: "616e395c87839a828e8b71a287736d40"
server_key_version: "kuduclusterkey@0"
I20260925 10:35:11.507408 13306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:11.573629 13306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:11.579113 13306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:11.579413 13306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:11.580521 13306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:11.582090 13306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:11.582198 13306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:11.582330 13306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:11.582415 13306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 10:35:11.645994 13306 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44351
I20260925 10:35:11.646047 13465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44351 every 8 connection(s)
I20260925 10:35:11.648041 13306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:35:11.656243 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 13306
I20260925 10:35:11.657021 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33909/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45187
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39895
--webserver_port=39981
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33909/kms with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:35:11 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:11.699388 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:11.658139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39895 (local address 127.0.97.1:52503)
0925 10:35:11.662458 (+ 4319us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:11.662504 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:11.664799 (+ 2295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:11.668572 (+ 3773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:11.668583 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:11.669205 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:11.669992 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:11.670017 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:11.672978 (+ 2961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:11.672984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:11.675031 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:11.675046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:11.676505 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:11.676512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:11.676793 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:11.677551 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:11.677579 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:11.680445 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:11.692113 (+ 11668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:11.692123 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:11.692140 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:11.692745 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:11.695442 (+ 2697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:11.695448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:11.695452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:11.695717 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:11.696775 (+ 1058us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:11.696784 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:11.697193 (+ 409us) client_negotiation.cc:769] Sending connection context
0925 10:35:11.697458 (+ 265us) client_negotiation.cc:240] Negotiation successful
0925 10:35:11.698007 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3994,"thread_start_us":3612,"threads_started":1}
I20260925 10:35:11.702618 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:11.657939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52503 (local address 127.0.97.62:39895)
0925 10:35:11.666930 (+ 8991us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:11.666949 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:11.667000 (+ 51us) server_negotiation.cc:407] Connection header received
0925 10:35:11.667467 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:11.667487 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:11.667961 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:11.670527 (+ 2566us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:11.670905 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:11.672773 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:11.675230 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:11.676214 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:11.677263 (+ 1049us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:11.680728 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:11.680737 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:11.680756 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:11.680804 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:11.691919 (+ 11115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:11.692956 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:11.692965 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:11.692973 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:11.695287 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:11.695865 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:11.695870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:11.695874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:11.696611 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:11.700711 (+ 4100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:11.700973 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 10:35:11.701371 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8723,"thread_start_us":8428,"threads_started":1}
I20260925 10:35:11.706177 13466 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39895
I20260925 10:35:11.706497 13466 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:11.707281 13466 heartbeater.cc:511] Master 127.0.97.62:39895 requested a full tablet report, sending...
W20260925 10:35:11.710284 13466 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39895 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:35:11.710515 13466 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:11.710764 13466 heartbeater.cc:511] Master 127.0.97.62:39895 requested a full tablet report, sending...
I20260925 10:35:11.712056 13466 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:11.712293 13466 heartbeater.cc:511] Master 127.0.97.62:39895 requested a full tablet report, sending...
W20260925 10:35:11.713359 13466 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:35:12.095968 13470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:12.096201 13470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:12.096341 13470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:12.105437 13470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:12.105985 13470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:12.106241 13470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:35:12.117877 13470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33909/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39981
--tserver_master_addrs=127.0.97.62:39895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:12.119575 13470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:12.121249 13470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:12.137727 13484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:12.137718 13485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:12.145646 13470 server_base.cc:1034] running on GCE node
W20260925 10:35:12.145829 13488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:12.146646 13470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:12.147872 13470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:12.149140 13470 hybrid_clock.cc:648] HybridClock initialized: now 1790332512149073 us; error 56 us; skew 500 ppm
I20260925 10:35:12.149524 13470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:12 dist-test-slave-dxdd krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:12.159866 13470 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:35:12.162429 13470 webserver.cc:460] Webserver started at http://127.0.97.2:39981/ using document root <none> and password file <none>
I20260925 10:35:12.163201 13470 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:12.163357 13470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:35:12 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260925 10:35:12.198912 13470 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.005s
I20260925 10:35:12.202560 13497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:12.204478 13470 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 10:35:12.204612 13470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "f01981365d43439a9a3810d94f455550"
format_stamp: "Formatted at 2026-09-25 10:35:09 on dist-test-slave-dxdd"
server_key: "80b6f461dca442ecb3ac54779a97b253"
server_key_iv: "730e757bb0b99b9a0434e2ed9f0d70dd"
server_key_version: "kuduclusterkey@0"
I20260925 10:35:12.205544 13470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:12.235909 13470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:12.240825 13470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:12.241124 13470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:12.242192 13470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:12.243844 13470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:12.243961 13470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:12.244108 13470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:12.244202 13470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:35:12 dist-test-slave-dxdd krb5kdc[12257](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790332512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:12.299086 13609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45187 every 8 connection(s)
I20260925 10:35:12.299417 13470 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45187
I20260925 10:35:12.301272 13470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:35:12.304762 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 13470
I20260925 10:35:12.305773 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33909/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36127
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39895
--webserver_port=41363
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33909/kms with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:35:12 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:12.345728 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:12.306393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35057 (local address 127.0.97.62:39895)
0925 10:35:12.321638 (+ 15245us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:12.321645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:12.326349 (+ 4704us) server_negotiation.cc:407] Connection header received
0925 10:35:12.327585 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:12.327591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:12.327680 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:12.327762 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:12.330061 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:12.331282 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:12.333556 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:12.334186 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:12.334282 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:12.338617 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:12.338623 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:12.338628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:12.338665 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:12.342214 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:12.343075 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:12.343081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:12.343085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:12.343233 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:12.343727 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:12.343732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:12.343735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:12.344052 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:12.344173 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:12.345121 (+ 948us) server_negotiation.cc:299] Negotiation successful
0925 10:35:12.345332 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15106,"thread_start_us":15245,"threads_started":1}
I20260925 10:35:12.348127 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:12.306954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39895 (local address 127.0.97.2:35057)
0925 10:35:12.323665 (+ 16711us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:12.323721 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:12.326213 (+ 2492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:12.328316 (+ 2103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:12.328328 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:12.328938 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:12.329833 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:12.329859 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:12.331485 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:12.331490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:12.333399 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:12.333413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:12.334662 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:12.334669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:12.334935 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:12.335703 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:12.335738 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:12.338381 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:12.342391 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:12.342400 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:12.342415 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:12.342931 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:12.343375 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:12.343379 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:12.343383 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:12.343605 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:12.344254 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:12.344262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:12.344641 (+ 379us) client_negotiation.cc:769] Sending connection context
0925 10:35:12.346271 (+ 1630us) client_negotiation.cc:240] Negotiation successful
0925 10:35:12.346819 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15023,"thread_start_us":14803,"threads_started":1}
I20260925 10:35:12.350046 13610 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39895
I20260925 10:35:12.350361 13610 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:12.351151 13610 heartbeater.cc:511] Master 127.0.97.62:39895 requested a full tablet report, sending...
I20260925 10:35:12.353003 13269 ts_manager.cc:194] Registered new tserver with Master: f01981365d43439a9a3810d94f455550 (127.0.97.2:45187)
I20260925 10:35:12.355154 13269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35057
I20260925 10:35:12.713518 13466 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:12.713985 13466 heartbeater.cc:511] Master 127.0.97.62:39895 requested a full tablet report, sending...
I20260925 10:35:12.715348 13269 ts_manager.cc:194] Registered new tserver with Master: 782672c1801c4db1b897b0867a6cd120 (127.0.97.1:44351)
I20260925 10:35:12.716542 13269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52503
Sep 25 10:35:12 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260925 10:35:12.774215 13614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:12.774658 13614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:12.774829 13614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:12.788357 13614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:12.788514 13614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:12.788677 13614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:35:12.809218 13614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33909/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41363
--tserver_master_addrs=127.0.97.62:39895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:12.811134 13614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:12.812772 13614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:12.855129 13620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:12.859848 13621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:12.912210 13614 server_base.cc:1034] running on GCE node
W20260925 10:35:12.913064 13623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:12.913861 13614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:12.928319 13614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:12.929610 13614 hybrid_clock.cc:648] HybridClock initialized: now 1790332512929509 us; error 99 us; skew 500 ppm
I20260925 10:35:12.930034 13614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:12 dist-test-slave-dxdd krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:12.934268 13614 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:35:12.965713 13614 webserver.cc:460] Webserver started at http://127.0.97.3:41363/ using document root <none> and password file <none>
I20260925 10:35:12.966883 13614 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:12.967103 13614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:35:12 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260925 10:35:13.000131 13614 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.002s
I20260925 10:35:13.003691 13631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:13.005368 13614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:35:13.005524 13614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "98b8c7d50b4843559ad50b51b61acc49"
format_stamp: "Formatted at 2026-09-25 10:35:09 on dist-test-slave-dxdd"
server_key: "a16aeed93f4cc0b4e07d43274fd4bbd7"
server_key_iv: "82606f0dc97acf6271627545b5019c24"
server_key_version: "kuduclusterkey@0"
I20260925 10:35:13.006747 13614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:13.037743 13614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:13.043138 13614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:13.043467 13614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:13.044588 13614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:13.054049 13614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:13.054198 13614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 10:35:13.054352 13614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:13.054450 13614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:35:13 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260925 10:35:13.359565 13610 heartbeater.cc:503] Master 127.0.97.62:39895 was elected leader, sending a full tablet report...
I20260925 10:35:13.381784 13614 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36127
I20260925 10:35:13.381866 13751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36127 every 8 connection(s)
I20260925 10:35:13.383646 13614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:35:13.392570 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 13614
Sep 25 10:35:13 dist-test-slave-dxdd krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:13.425760 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:13.398672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53456 (local address 127.0.97.62:39895)
0925 10:35:13.405184 (+ 6512us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:13.405189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:13.407084 (+ 1895us) server_negotiation.cc:407] Connection header received
0925 10:35:13.407157 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:13.407162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:13.407229 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:13.407320 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:13.409427 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.410608 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.412491 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.413133 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.413249 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:13.417176 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:13.417182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:13.417187 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:13.417221 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:13.422115 (+ 4894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:13.422925 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:13.422930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:13.422935 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:13.423079 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:13.423500 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:13.423506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:13.423509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:13.423868 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:13.424114 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:13.425187 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0925 10:35:13.425389 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6378,"thread_start_us":6516,"threads_started":1}
I20260925 10:35:13.427408 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:13.386663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42863 (local address 127.0.97.62:39895)
0925 10:35:13.398727 (+ 12064us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:13.398734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:13.403335 (+ 4601us) server_negotiation.cc:407] Connection header received
0925 10:35:13.403428 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:13.403437 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:13.403572 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:13.403708 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:13.409220 (+ 5512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.410602 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.413660 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.414311 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.414430 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:13.418995 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:13.419001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:13.419005 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:13.419050 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:13.422805 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:13.423955 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:13.423960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:13.423964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:13.424097 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:13.425249 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:13.425255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:13.425258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:13.425593 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:13.425727 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:13.426851 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0925 10:35:13.427054 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11948,"thread_start_us":11820,"threads_started":1}
I20260925 10:35:13.430120 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:13.387143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39895 (local address 127.0.97.3:42863)
0925 10:35:13.399422 (+ 12279us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:13.399488 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:13.402872 (+ 3384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:13.407098 (+ 4226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:13.407111 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:13.407749 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:13.408688 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:13.408716 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.410823 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.410829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:13.413484 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:13.413511 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.414765 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.414772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:13.415046 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:13.415757 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:13.415786 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:13.418716 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:13.423134 (+ 4418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:13.423143 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:13.423159 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:13.423805 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:13.424235 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:13.424239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:13.424243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:13.424914 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:13.425826 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:13.425834 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:13.426253 (+ 419us) client_negotiation.cc:769] Sending connection context
0925 10:35:13.427870 (+ 1617us) client_negotiation.cc:240] Negotiation successful
0925 10:35:13.428647 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11881,"thread_start_us":11653,"threads_started":1}
I20260925 10:35:13.431511 13752 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39895
I20260925 10:35:13.431847 13752 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:13.432703 13752 heartbeater.cc:511] Master 127.0.97.62:39895 requested a full tablet report, sending...
I20260925 10:35:13.434281 13269 ts_manager.cc:194] Registered new tserver with Master: 98b8c7d50b4843559ad50b51b61acc49 (127.0.97.3:36127)
I20260925 10:35:13.435352 13269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42863
I20260925 10:35:13.440976 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:35:13.467094 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:13.444340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53470 (local address 127.0.97.62:39895)
0925 10:35:13.444497 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:13.444502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:13.450942 (+ 6440us) server_negotiation.cc:407] Connection header received
0925 10:35:13.451219 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:13.451224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:13.451317 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:13.451414 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:13.453486 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.454489 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.456475 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.457052 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.457176 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:13.460694 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:13.460702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:13.460707 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:13.460763 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:13.464105 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:13.464962 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:13.464967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:13.464970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:13.465127 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:13.465530 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:13.465534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:13.465537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:13.465884 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:13.466133 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:13.466539 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 10:35:13.466741 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:35:13.472852 13269 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:35:13.483521 13269 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:35:13.564472 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:13.501479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49578 (local address 127.0.97.1:44351)
0925 10:35:13.527141 (+ 25662us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:13.527153 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:13.551361 (+ 24208us) server_negotiation.cc:407] Connection header received
0925 10:35:13.551893 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:13.551902 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:13.552188 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:13.552365 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:13.553388 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.555041 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.559845 (+ 4804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.561596 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.561712 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:13.561952 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:13.563953 (+ 2001us) server_negotiation.cc:299] Negotiation successful
0925 10:35:13.564130 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25396,"thread_start_us":25669,"threads_started":1}
I20260925 10:35:13.565555 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:13.501151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44351 (local address 127.0.0.1:49578)
0925 10:35:13.551228 (+ 50077us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:13.551248 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:13.551673 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:13.552728 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:13.552735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:13.552764 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:13.553145 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:13.553159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.555218 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.555223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:13.557559 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:13.557574 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.562578 (+ 5004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.562584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:13.562782 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:13.562829 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 10:35:13.565005 (+ 2176us) client_negotiation.cc:240] Negotiation successful
0925 10:35:13.565198 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49958,"thread_start_us":50243,"threads_started":1}
I20260925 10:35:13.570390 13401 tablet_service.cc:1467] Processing CreateTablet for tablet 832b0617f5b8421eadbefe1ce4bce645 (DEFAULT_TABLE table=test-table [id=64cfb2dd46d3452db31ddcdc2cf01c33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:35:13.571966 13401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832b0617f5b8421eadbefe1ce4bce645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:13.572223 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:13.499149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36127 (local address 127.0.0.1:56874)
0925 10:35:13.545998 (+ 46849us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:13.546059 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:13.546294 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:13.558753 (+ 12459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:13.558758 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:13.558783 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:13.559106 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:13.559117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.567827 (+ 8710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.567831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:13.569672 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:13.569685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.571483 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.571491 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:13.571649 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:13.571720 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 10:35:13.571786 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 10:35:13.571897 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46453,"mutex_wait_us":3,"thread_start_us":61953,"threads_started":1}
I20260925 10:35:13.573316 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:13.499474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.0.97.3:36127)
0925 10:35:13.558082 (+ 58608us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:13.558090 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:13.558123 (+ 33us) server_negotiation.cc:407] Connection header received
0925 10:35:13.558207 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:13.558214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:13.558532 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:13.559290 (+ 758us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:13.559650 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.567671 (+ 8021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.569862 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.571208 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.572536 (+ 1328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:13.572751 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:13.572812 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 10:35:13.572986 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58437,"thread_start_us":58318,"threads_started":1}
I20260925 10:35:13.576778 13680 tablet_service.cc:1467] Processing CreateTablet for tablet 832b0617f5b8421eadbefe1ce4bce645 (DEFAULT_TABLE table=test-table [id=64cfb2dd46d3452db31ddcdc2cf01c33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:35:13.578176 13680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832b0617f5b8421eadbefe1ce4bce645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:13.578634 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:13.502050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.0.97.2:45187)
0925 10:35:13.546756 (+ 44706us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:13.546766 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:13.562126 (+ 15360us) server_negotiation.cc:407] Connection header received
0925 10:35:13.562229 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:13.562236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:13.562504 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:13.562695 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:13.563473 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.565031 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.568592 (+ 3561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.569928 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.570033 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:13.570236 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:13.578103 (+ 7867us) server_negotiation.cc:299] Negotiation successful
0925 10:35:13.578298 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44531,"thread_start_us":45616,"threads_started":1}
I20260925 10:35:13.579385 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:13.501513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45187 (local address 127.0.0.1:41968)
0925 10:35:13.560888 (+ 59375us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:13.560908 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:13.561033 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:13.562940 (+ 1907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:13.562945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:13.562969 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:13.563273 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:13.563284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.566412 (+ 3128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.566417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:13.568417 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:13.568430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.571526 (+ 3096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.571534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:13.577870 (+ 6336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:13.577905 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:35:13.578867 (+ 962us) client_negotiation.cc:240] Negotiation successful
0925 10:35:13.579009 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59249,"mutex_wait_us":6136,"thread_start_us":45280,"threads_started":1}
I20260925 10:35:13.582999 13545 tablet_service.cc:1467] Processing CreateTablet for tablet 832b0617f5b8421eadbefe1ce4bce645 (DEFAULT_TABLE table=test-table [id=64cfb2dd46d3452db31ddcdc2cf01c33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:35:13.584383 13545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832b0617f5b8421eadbefe1ce4bce645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:13.597131 13773 tablet_bootstrap.cc:492] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120: Bootstrap starting.
I20260925 10:35:13.600512 13773 tablet_bootstrap.cc:654] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:13.601672 13773 log.cc:824] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:13.606668 13774 tablet_bootstrap.cc:492] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49: Bootstrap starting.
I20260925 10:35:13.608701 13775 tablet_bootstrap.cc:492] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550: Bootstrap starting.
I20260925 10:35:13.612393 13774 tablet_bootstrap.cc:654] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:13.613656 13774 log.cc:824] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:13.613806 13773 tablet_bootstrap.cc:492] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120: No bootstrap required, opened a new log
I20260925 10:35:13.615388 13775 tablet_bootstrap.cc:654] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:13.615724 13773 ts_tablet_manager.cc:1397] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20260925 10:35:13.624032 13774 tablet_bootstrap.cc:492] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49: No bootstrap required, opened a new log
I20260925 10:35:13.624375 13774 ts_tablet_manager.cc:1397] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.009s
I20260925 10:35:13.626418 13773 raft_consensus.cc:348] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b8c7d50b4843559ad50b51b61acc49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "782672c1801c4db1b897b0867a6cd120" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44351 } } peers { permanent_uuid: "f01981365d43439a9a3810d94f455550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45187 } }
I20260925 10:35:13.627364 13775 log.cc:824] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:13.633164 13773 raft_consensus.cc:374] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:13.633519 13773 raft_consensus.cc:729] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782672c1801c4db1b897b0867a6cd120, State: Initialized, Role: FOLLOWER
I20260925 10:35:13.634696 13773 consensus_queue.cc:260] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b8c7d50b4843559ad50b51b61acc49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "782672c1801c4db1b897b0867a6cd120" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44351 } } peers { permanent_uuid: "f01981365d43439a9a3810d94f455550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45187 } }
I20260925 10:35:13.638175 13775 tablet_bootstrap.cc:492] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550: No bootstrap required, opened a new log
I20260925 10:35:13.638581 13775 ts_tablet_manager.cc:1397] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.000s
I20260925 10:35:13.638307 13774 raft_consensus.cc:348] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b8c7d50b4843559ad50b51b61acc49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "782672c1801c4db1b897b0867a6cd120" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44351 } } peers { permanent_uuid: "f01981365d43439a9a3810d94f455550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45187 } }
I20260925 10:35:13.638870 13774 raft_consensus.cc:374] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:13.638990 13774 raft_consensus.cc:729] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98b8c7d50b4843559ad50b51b61acc49, State: Initialized, Role: FOLLOWER
I20260925 10:35:13.639685 13774 consensus_queue.cc:260] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b8c7d50b4843559ad50b51b61acc49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "782672c1801c4db1b897b0867a6cd120" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44351 } } peers { permanent_uuid: "f01981365d43439a9a3810d94f455550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45187 } }
I20260925 10:35:13.644902 13466 heartbeater.cc:503] Master 127.0.97.62:39895 was elected leader, sending a full tablet report...
I20260925 10:35:13.650689 13773 ts_tablet_manager.cc:1428] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120: Time spent starting tablet: real 0.035s user 0.006s sys 0.008s
I20260925 10:35:13.651988 13775 raft_consensus.cc:348] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b8c7d50b4843559ad50b51b61acc49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "782672c1801c4db1b897b0867a6cd120" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44351 } } peers { permanent_uuid: "f01981365d43439a9a3810d94f455550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45187 } }
I20260925 10:35:13.652449 13775 raft_consensus.cc:374] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:13.652580 13775 raft_consensus.cc:729] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f01981365d43439a9a3810d94f455550, State: Initialized, Role: FOLLOWER
I20260925 10:35:13.653276 13775 consensus_queue.cc:260] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b8c7d50b4843559ad50b51b61acc49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "782672c1801c4db1b897b0867a6cd120" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44351 } } peers { permanent_uuid: "f01981365d43439a9a3810d94f455550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45187 } }
I20260925 10:35:13.661249 13775 ts_tablet_manager.cc:1428] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550: Time spent starting tablet: real 0.023s user 0.012s sys 0.006s
W20260925 10:35:13.663247 13467 tablet.cc:2369] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:35:13.666795 13752 heartbeater.cc:503] Master 127.0.97.62:39895 was elected leader, sending a full tablet report...
I20260925 10:35:13.667644 13774 ts_tablet_manager.cc:1428] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49: Time spent starting tablet: real 0.043s user 0.015s sys 0.000s
I20260925 10:35:13.815469 13781 raft_consensus.cc:482] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:35:13.815785 13781 raft_consensus.cc:504] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b8c7d50b4843559ad50b51b61acc49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "782672c1801c4db1b897b0867a6cd120" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44351 } } peers { permanent_uuid: "f01981365d43439a9a3810d94f455550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45187 } }
I20260925 10:35:13.817814 13781 leader_election.cc:288] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98b8c7d50b4843559ad50b51b61acc49 (127.0.97.3:36127), 782672c1801c4db1b897b0867a6cd120 (127.0.97.1:44351)
I20260925 10:35:13.828171 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:13.818502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41949 (local address 127.0.97.1:44351)
0925 10:35:13.818696 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:13.819726 (+ 1030us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:13.820836 (+ 1110us) server_negotiation.cc:407] Connection header received
0925 10:35:13.821037 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:13.821045 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:13.821131 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:13.821182 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:13.821897 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.822948 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.825751 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.826991 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.827078 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:13.827150 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:13.827720 (+ 570us) server_negotiation.cc:299] Negotiation successful
0925 10:35:13.827854 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 10:35:13.828125 13611 tablet.cc:2369] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:35:13.828174 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:13.818926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33195 (local address 127.0.97.3:36127)
0925 10:35:13.819208 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:13.819216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:13.820399 (+ 1183us) server_negotiation.cc:407] Connection header received
0925 10:35:13.820702 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:13.820709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:13.820803 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:13.820899 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:13.821469 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.822437 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.825209 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.827267 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.827397 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:13.827485 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:13.827804 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 10:35:13.827915 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:35:13.828787 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:13.818215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44351 (local address 127.0.97.2:41949)
0925 10:35:13.820717 (+ 2502us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:13.820734 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:13.820870 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:13.821321 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:13.821324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:13.821342 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:13.821725 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:13.821737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.823091 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.823096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:13.825572 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:13.825586 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.827315 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.827319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:13.827527 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:13.827566 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 10:35:13.828337 (+ 771us) client_negotiation.cc:240] Negotiation successful
0925 10:35:13.828468 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2400,"thread_start_us":1774,"threads_started":1}
I20260925 10:35:13.829258 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:13.818161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36127 (local address 127.0.97.2:33195)
0925 10:35:13.820280 (+ 2119us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:13.820303 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:13.820521 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:13.820924 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:13.820929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:13.820963 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:13.821295 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:13.821305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.822558 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.822567 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:13.825041 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:13.825055 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.827478 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.827484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:13.827641 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:13.827685 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 10:35:13.827766 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 10:35:13.828882 (+ 1116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1981,"thread_start_us":2271,"threads_started":1}
I20260925 10:35:13.830062 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832b0617f5b8421eadbefe1ce4bce645" candidate_uuid: "f01981365d43439a9a3810d94f455550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782672c1801c4db1b897b0867a6cd120" is_pre_election: true
I20260925 10:35:13.830175 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832b0617f5b8421eadbefe1ce4bce645" candidate_uuid: "f01981365d43439a9a3810d94f455550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98b8c7d50b4843559ad50b51b61acc49" is_pre_election: true
I20260925 10:35:13.830670 13700 raft_consensus.cc:2454] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f01981365d43439a9a3810d94f455550 in term 0.
I20260925 10:35:13.830655 13421 raft_consensus.cc:2454] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f01981365d43439a9a3810d94f455550 in term 0.
I20260925 10:35:13.831310 13499 leader_election.cc:302] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 782672c1801c4db1b897b0867a6cd120, f01981365d43439a9a3810d94f455550; no voters:
I20260925 10:35:13.831815 13781 raft_consensus.cc:2781] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:35:13.832054 13781 raft_consensus.cc:482] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:35:13.832185 13781 raft_consensus.cc:3037] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:13.835367 13781 raft_consensus.cc:504] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b8c7d50b4843559ad50b51b61acc49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "782672c1801c4db1b897b0867a6cd120" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44351 } } peers { permanent_uuid: "f01981365d43439a9a3810d94f455550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45187 } }
I20260925 10:35:13.836019 13781 leader_election.cc:288] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [CANDIDATE]: Term 1 election: Requested vote from peers 98b8c7d50b4843559ad50b51b61acc49 (127.0.97.3:36127), 782672c1801c4db1b897b0867a6cd120 (127.0.97.1:44351)
I20260925 10:35:13.836438 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832b0617f5b8421eadbefe1ce4bce645" candidate_uuid: "f01981365d43439a9a3810d94f455550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782672c1801c4db1b897b0867a6cd120"
I20260925 10:35:13.836463 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832b0617f5b8421eadbefe1ce4bce645" candidate_uuid: "f01981365d43439a9a3810d94f455550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98b8c7d50b4843559ad50b51b61acc49"
I20260925 10:35:13.836647 13421 raft_consensus.cc:3037] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:13.836647 13700 raft_consensus.cc:3037] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:13.838862 13421 raft_consensus.cc:2454] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f01981365d43439a9a3810d94f455550 in term 1.
I20260925 10:35:13.839252 13499 leader_election.cc:302] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 782672c1801c4db1b897b0867a6cd120, f01981365d43439a9a3810d94f455550; no voters:
I20260925 10:35:13.839527 13781 raft_consensus.cc:2781] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:35:13.839745 13700 raft_consensus.cc:2454] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f01981365d43439a9a3810d94f455550 in term 1.
I20260925 10:35:13.840059 13781 raft_consensus.cc:686] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [term 1 LEADER]: Becoming Leader. State: Replica: f01981365d43439a9a3810d94f455550, State: Running, Role: LEADER
I20260925 10:35:13.840564 13781 consensus_queue.cc:237] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b8c7d50b4843559ad50b51b61acc49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "782672c1801c4db1b897b0867a6cd120" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44351 } } peers { permanent_uuid: "f01981365d43439a9a3810d94f455550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45187 } }
I20260925 10:35:13.845799 13269 catalog_manager.cc:5495] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 reported cstate change: term changed from 0 to 1, leader changed from <none> to f01981365d43439a9a3810d94f455550 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f01981365d43439a9a3810d94f455550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b8c7d50b4843559ad50b51b61acc49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "782672c1801c4db1b897b0867a6cd120" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f01981365d43439a9a3810d94f455550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45187 } health_report { overall_health: HEALTHY } } }
I20260925 10:35:13.871750 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:13.865768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.0.97.2:45187)
0925 10:35:13.866053 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:13.866059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:13.866076 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:35:13.866273 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:13.866277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:13.866434 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:13.866584 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:35:13.866937 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.867662 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.869389 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.869934 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.870056 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:13.870615 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:35:13.871210 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:35:13.871345 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:13.871444 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 10:35:13.871512 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260925 10:35:13.881494 13421 raft_consensus.cc:1261] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120 [term 1 FOLLOWER]: Refusing update from remote peer f01981365d43439a9a3810d94f455550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:35:13.881600 13700 raft_consensus.cc:1261] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49 [term 1 FOLLOWER]: Refusing update from remote peer f01981365d43439a9a3810d94f455550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:35:13.882642 13784 consensus_queue.cc:1035] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782672c1801c4db1b897b0867a6cd120" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:35:13.883261 13784 consensus_queue.cc:1035] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98b8c7d50b4843559ad50b51b61acc49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:35:13.886763 13753 tablet.cc:2369] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:35:13.904388 13788 mvcc.cc:205] Tried to move back new op lower bound from 7333201976848404480 to 7333201976697569280. Current Snapshot: MvccSnapshot[applied={T|T < 7333201976848404480}]
I20260925 10:35:13.919597 13269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:35:13.919845 13269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:35:13.923554 13269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3ec021f327d4aa5ad21ffb97f32ddeb: Sending DeleteTablet for 3 replicas of tablet 832b0617f5b8421eadbefe1ce4bce645
I20260925 10:35:13.924593 13680 tablet_service.cc:1514] Processing DeleteTablet for tablet 832b0617f5b8421eadbefe1ce4bce645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:35:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56874
I20260925 10:35:13.924970 13545 tablet_service.cc:1514] Processing DeleteTablet for tablet 832b0617f5b8421eadbefe1ce4bce645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:35:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41968
I20260925 10:35:13.926092 13401 tablet_service.cc:1514] Processing DeleteTablet for tablet 832b0617f5b8421eadbefe1ce4bce645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:35:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49578
I20260925 10:35:13.938807 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 13306
I20260925 10:35:13.960893 13797 tablet_replica.cc:330] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550: stopping tablet replica
I20260925 10:35:13.961583 13798 tablet_replica.cc:330] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120: stopping tablet replica
I20260925 10:35:13.963549 13798 raft_consensus.cc:2229] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:35:13.963840 13797 raft_consensus.cc:2229] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:35:13.964612 13797 raft_consensus.cc:2258] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:35:13.965215 13798 raft_consensus.cc:2258] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:35:13.968336 13797 ts_tablet_manager.cc:1905] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:35:13.969704 13798 ts_tablet_manager.cc:1905] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:35:13.977145 13798 ts_tablet_manager.cc:1918] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:35:13.982327 13796 tablet_replica.cc:330] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49: stopping tablet replica
I20260925 10:35:13.982836 13796 raft_consensus.cc:2229] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:35:13.983412 13796 raft_consensus.cc:2258] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:35:13.985461 13796 ts_tablet_manager.cc:1905] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:35:13.988766 13797 ts_tablet_manager.cc:1918] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:35:13.988926 13797 log.cc:1196] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/832b0617f5b8421eadbefe1ce4bce645
I20260925 10:35:13.989451 13797 ts_tablet_manager.cc:1939] T 832b0617f5b8421eadbefe1ce4bce645 P f01981365d43439a9a3810d94f455550: Deleting consensus metadata
I20260925 10:35:13.985467 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:13.940798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49586 (local address 127.0.97.1:44351)
0925 10:35:13.940956 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:13.940961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:13.960513 (+ 19552us) server_negotiation.cc:407] Connection header received
0925 10:35:13.960998 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:13.961006 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:13.961130 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:13.961237 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:13.970870 (+ 9633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.971941 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.973793 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:13.974417 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:13.974541 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:13.977517 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:13.977529 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:13.977545 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:13.977587 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:13.981433 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:13.982447 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:13.982456 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:13.982460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:13.982609 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:13.983114 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:13.983118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:13.983121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:13.983538 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:13.983829 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:13.984874 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0925 10:35:13.985091 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 10:35:13.990572 13798 log.cc:1196] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/832b0617f5b8421eadbefe1ce4bce645
I20260925 10:35:13.991174 13798 ts_tablet_manager.cc:1939] T 832b0617f5b8421eadbefe1ce4bce645 P 782672c1801c4db1b897b0867a6cd120: Deleting consensus metadata
I20260925 10:35:13.991288 13461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:13.993077 13255 catalog_manager.cc:4841] TS f01981365d43439a9a3810d94f455550 (127.0.97.2:45187): tablet 832b0617f5b8421eadbefe1ce4bce645 (table test-table [id=64cfb2dd46d3452db31ddcdc2cf01c33]) successfully deleted
I20260925 10:35:13.995317 13796 ts_tablet_manager.cc:1918] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:35:13.995502 13796 log.cc:1196] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/832b0617f5b8421eadbefe1ce4bce645
I20260925 10:35:13.995980 13796 ts_tablet_manager.cc:1939] T 832b0617f5b8421eadbefe1ce4bce645 P 98b8c7d50b4843559ad50b51b61acc49: Deleting consensus metadata
I20260925 10:35:13.997633 13253 catalog_manager.cc:4841] TS 98b8c7d50b4843559ad50b51b61acc49 (127.0.97.3:36127): tablet 832b0617f5b8421eadbefe1ce4bce645 (table test-table [id=64cfb2dd46d3452db31ddcdc2cf01c33]) successfully deleted
I20260925 10:35:14.134771 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 13306
I20260925 10:35:14.134943 13254 catalog_manager.cc:4841] TS 782672c1801c4db1b897b0867a6cd120 (127.0.97.1:44351): tablet 832b0617f5b8421eadbefe1ce4bce645 (table test-table [id=64cfb2dd46d3452db31ddcdc2cf01c33]) successfully deleted
I20260925 10:35:14.151542 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 13470
I20260925 10:35:14.167225 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:14.152496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.0.97.2:45187)
0925 10:35:14.152658 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:14.152662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:14.152678 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:35:14.152729 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:14.152733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:14.152797 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:14.152875 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:14.154503 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:14.155348 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:14.157279 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:14.157713 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:14.157820 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:14.160941 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:14.160954 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:14.160969 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:14.161009 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:14.164131 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:14.164950 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:14.164958 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:14.164960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:14.165101 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:14.165508 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:14.165511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:14.165513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:14.165916 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:14.166267 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:14.166680 (+ 413us) server_negotiation.cc:299] Negotiation successful
0925 10:35:14.166915 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:35:14.167928 13605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:14.282950 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 13470
I20260925 10:35:14.301707 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 13614
I20260925 10:35:14.314540 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:14.302410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.0.97.3:36127)
0925 10:35:14.302593 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:14.302600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:14.302667 (+ 67us) server_negotiation.cc:407] Connection header received
0925 10:35:14.302903 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:14.302907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:14.302969 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:14.303071 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:14.304325 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:14.305063 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:14.306422 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:14.306828 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:14.306924 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:14.309257 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:14.309270 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:14.309285 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:14.309314 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:14.312223 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:14.312803 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:14.312810 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:14.312812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:14.312922 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:14.313236 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:14.313239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:14.313241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:14.313565 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:14.313790 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:14.314019 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 10:35:14.314166 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:35:14.315141 13747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:14.402164 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 13614
I20260925 10:35:14.418422 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 13236
I20260925 10:35:14.419584 13299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:14.529320 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 13236
I20260925 10:35:14.551003 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 10:35:14.886270 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 10:35:14.886559 388 mini_ranger.cc:66] Stopping Ranger...
I20260925 10:35:15.232334 388 mini_ranger.cc:68] Stopped Ranger
2026-09-25 10:35:15.232 UTC [12287] LOG: received smart shutdown request
2026-09-25 10:35:15.235 UTC [12287] LOG: background worker "logical replication launcher" (PID 12296) exited with exit code 1
2026-09-25 10:35:15.236 UTC [12291] LOG: shutting down
2026-09-25 10:35:15.403 UTC [12287] LOG: database system is shut down
2026-09-25T10:35:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52384 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:35:15 dist-test-slave-dxdd krb5kdc[13812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:35:15 dist-test-slave-dxdd krb5kdc[13812](info): set up 2 sockets
Sep 25 10:35:15 dist-test-slave-dxdd krb5kdc[13812](info): commencing operation
krb5kdc: starting...
W20260925 10:35:16.610471 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:35:16 dist-test-slave-dxdd krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:35:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:35:16.823236 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36197
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36197,127.0.97.61:39577,127.0.97.60:39063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:35:17.140443 13825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:17.140702 13825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:17.140888 13825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:17.147729 13825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:35:17.147846 13825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:17.147929 13825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:17.148015 13825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:35:17.148085 13825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:35:17.157181 13825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36197,127.0.97.61:39577,127.0.97.60:39063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:17.158465 13825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:17.159744 13825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:17.171270 13831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:17.171360 13833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:17.171386 13830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:17.172298 13825 server_base.cc:1034] running on GCE node
I20260925 10:35:17.173069 13825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:17.174468 13825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:17.175661 13825 hybrid_clock.cc:648] HybridClock initialized: now 1790332517175612 us; error 48 us; skew 500 ppm
I20260925 10:35:17.176008 13825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:17 dist-test-slave-dxdd krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:17.184656 13825 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:35:17.186028 13825 webserver.cc:460] Webserver started at http://127.0.97.62:43335/ using document root <none> and password file <none>
I20260925 10:35:17.186760 13825 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:17.186859 13825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:17.187136 13825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:35:17.190111 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "b569e12627ab4555acb04172ef255d51"
format_stamp: "Formatted at 2026-09-25 10:35:17 on dist-test-slave-dxdd"
server_key: "d0c55ceb57744c3088ae1c1dd9f5a9cd"
server_key_iv: "47dc9cb789edebf9cb960ad8908f82b6"
server_key_version: "encryptionkey@0"
I20260925 10:35:17.190896 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "b569e12627ab4555acb04172ef255d51"
format_stamp: "Formatted at 2026-09-25 10:35:17 on dist-test-slave-dxdd"
server_key: "d0c55ceb57744c3088ae1c1dd9f5a9cd"
server_key_iv: "47dc9cb789edebf9cb960ad8908f82b6"
server_key_version: "encryptionkey@0"
I20260925 10:35:17.195731 13825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 10:35:17.198879 13840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:17.200169 13825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:35:17.200349 13825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "b569e12627ab4555acb04172ef255d51"
format_stamp: "Formatted at 2026-09-25 10:35:17 on dist-test-slave-dxdd"
server_key: "d0c55ceb57744c3088ae1c1dd9f5a9cd"
server_key_iv: "47dc9cb789edebf9cb960ad8908f82b6"
server_key_version: "encryptionkey@0"
I20260925 10:35:17.200541 13825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:17.212894 13825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:17.216538 13825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:17.216820 13825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:17.235183 13825 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36197
I20260925 10:35:17.235208 13891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36197 every 8 connection(s)
I20260925 10:35:17.236516 13825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:35:17.240092 13892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:17.242538 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 13825
I20260925 10:35:17.242765 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:35:17.243398 388 external_mini_cluster.cc:1428] Setting key faef76c17d5e661aa2843637f3df83e7
I20260925 10:35:17.246551 13892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:35:17 dist-test-slave-dxdd krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260925 10:35:17.268795 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.249747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.0.97.62:36197)
0925 10:35:17.251434 (+ 1687us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:17.251442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:17.251492 (+ 50us) server_negotiation.cc:407] Connection header received
0925 10:35:17.252915 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:17.252920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:17.253134 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:17.253339 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:17.254593 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:17.255736 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:17.257123 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:17.257693 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:17.257888 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:17.261193 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:17.261201 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:17.261212 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:17.261240 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:17.264540 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:17.265191 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:17.265203 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:17.265205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:17.265334 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:17.265732 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:17.265736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:17.265741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:17.266217 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:17.266466 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:17.267003 (+ 537us) server_negotiation.cc:299] Negotiation successful
0925 10:35:17.267389 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":1443,"thread_start_us":1234,"threads_started":1}
I20260925 10:35:17.268827 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.249747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36197 (local address 127.0.0.1:54034)
0925 10:35:17.250618 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.250663 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:17.252547 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:17.253351 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:17.253357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:17.254027 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:17.254408 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:17.254428 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:17.255893 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:17.255898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:17.256988 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:17.257000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:17.257929 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:17.257936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:17.258103 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:17.258817 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:17.258839 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:17.260967 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:17.264659 (+ 3692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:17.264678 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:17.264690 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:17.265050 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:17.265444 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:17.265451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:17.265457 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:17.265615 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:17.266374 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:17.266383 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:17.266609 (+ 226us) client_negotiation.cc:769] Sending connection context
0925 10:35:17.266861 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 10:35:17.267507 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":193,"thread_start_us":391,"threads_started":1}
I20260925 10:35:17.271991 13892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:35:17.273097 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.272711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39577 (local address 127.0.0.1:52950)
0925 10:35:17.272872 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.272986 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 10:35:17.273363 13841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39577: connect: Connection refused (error 111)
I20260925 10:35:17.274832 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.274515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39577 (local address 127.0.0.1:52962)
0925 10:35:17.274662 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.274721 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 10:35:17.275033 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39577: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39577: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260925 10:35:17.290714 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39577
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36197,127.0.97.61:39577,127.0.97.60:39063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:35:17.324321 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } attempt: 1
I20260925 10:35:17.325532 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.324987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39577 (local address 127.0.0.1:52976)
0925 10:35:17.325239 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.325349 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 10:35:17.326609 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.326212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39577 (local address 127.0.0.1:52986)
0925 10:35:17.326381 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.326463 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:35:17.326800 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39577: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39577: connect: Connection refused (error 111)
I20260925 10:35:17.395080 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } attempt: 2
I20260925 10:35:17.396373 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.395809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39577 (local address 127.0.0.1:52998)
0925 10:35:17.396073 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.396195 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 10:35:17.397480 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.397040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39577 (local address 127.0.0.1:53010)
0925 10:35:17.397219 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.397318 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 10:35:17.397693 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39577: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39577: connect: Connection refused (error 111)
I20260925 10:35:17.488948 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } attempt: 3
I20260925 10:35:17.490224 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.489746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39577 (local address 127.0.0.1:53012)
0925 10:35:17.489945 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.490051 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 10:35:17.491285 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.490870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39577 (local address 127.0.0.1:53020)
0925 10:35:17.491011 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.491126 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:35:17.491464 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39577: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39577: connect: Connection refused (error 111)
W20260925 10:35:17.603427 13899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:17.603714 13899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:17.603899 13899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:17.610703 13899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:35:17.610810 13899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:17.610922 13899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:17.611020 13899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:35:17.611101 13899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:35:17.620246 13899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36197,127.0.97.61:39577,127.0.97.60:39063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:17.621475 13899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:17.622718 13899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:17.634121 13907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:17.634158 13904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:17.634429 13905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:17.634670 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } attempt: 4
I20260925 10:35:17.635082 13899 server_base.cc:1034] running on GCE node
I20260925 10:35:17.635701 13899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:17.635743 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.635254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39577 (local address 127.0.0.1:53034)
0925 10:35:17.635531 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.635620 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260925 10:35:17.636749 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.636419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39577 (local address 127.0.0.1:53040)
0925 10:35:17.636556 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.636640 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 10:35:17.637014 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39577: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39577: connect: Connection refused (error 111)
I20260925 10:35:17.637288 13899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:17.638504 13899 hybrid_clock.cc:648] HybridClock initialized: now 1790332517638462 us; error 59 us; skew 500 ppm
I20260925 10:35:17.638852 13899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:17 dist-test-slave-dxdd krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:17.641896 13899 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260925 10:35:17.643224 13899 webserver.cc:460] Webserver started at http://127.0.97.61:41551/ using document root <none> and password file <none>
I20260925 10:35:17.643832 13899 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:17.643924 13899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:17.644158 13899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:35:17.646978 13899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/data/instance:
uuid: "2229f7236e504be9bb6d48dbf732883a"
format_stamp: "Formatted at 2026-09-25 10:35:17 on dist-test-slave-dxdd"
server_key: "374e549a6c427e4a19f5cd8014b113e8"
server_key_iv: "80630ceff5d1891ae42f12d93382c849"
server_key_version: "encryptionkey@0"
I20260925 10:35:17.647702 13899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/wal/instance:
uuid: "2229f7236e504be9bb6d48dbf732883a"
format_stamp: "Formatted at 2026-09-25 10:35:17 on dist-test-slave-dxdd"
server_key: "374e549a6c427e4a19f5cd8014b113e8"
server_key_iv: "80630ceff5d1891ae42f12d93382c849"
server_key_version: "encryptionkey@0"
I20260925 10:35:17.652498 13899 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:35:17.655699 13914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:17.656977 13899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 10:35:17.657102 13899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/wal
uuid: "2229f7236e504be9bb6d48dbf732883a"
format_stamp: "Formatted at 2026-09-25 10:35:17 on dist-test-slave-dxdd"
server_key: "374e549a6c427e4a19f5cd8014b113e8"
server_key_iv: "80630ceff5d1891ae42f12d93382c849"
server_key_version: "encryptionkey@0"
I20260925 10:35:17.657250 13899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:17.675149 13899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:17.679329 13899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:17.679556 13899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:17.697592 13899 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39577
I20260925 10:35:17.697625 13965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39577 every 8 connection(s)
I20260925 10:35:17.698884 13899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/data/info.pb
I20260925 10:35:17.700361 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 13899
I20260925 10:35:17.700555 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-1/wal/instance
I20260925 10:35:17.700966 388 external_mini_cluster.cc:1428] Setting key 1d647eb04668546033dfe7aa3e9b39c2
I20260925 10:35:17.702633 13966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:17.709183 13966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 25 10:35:17 dist-test-slave-dxdd krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:17.729568 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.711865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.0.97.62:36197)
0925 10:35:17.712080 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:17.712086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:17.713228 (+ 1142us) server_negotiation.cc:407] Connection header received
0925 10:35:17.715099 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:17.715104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:17.715192 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:17.715315 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:17.716735 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:17.717423 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:17.719067 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:17.719523 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:17.719662 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:17.723592 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:17.723597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:17.723601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:17.723634 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:17.726147 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:17.727113 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:17.727117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:17.727119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:17.727236 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:17.727698 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:17.727701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:17.727703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:17.728069 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:17.728222 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:17.728876 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 10:35:17.729140 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 10:35:17.730087 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.712237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36197 (local address 127.0.0.1:54038)
0925 10:35:17.713040 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.713094 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:17.714830 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:17.715538 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:17.715545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:17.716033 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:17.716558 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:17.716575 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:17.717586 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:17.717590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:17.718920 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:17.718933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:17.719744 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:17.719750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:17.719969 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:17.720643 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:17.720664 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:17.723352 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:17.726324 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:17.726336 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:17.726354 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:17.726987 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:17.727346 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:17.727354 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:17.727360 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:17.727588 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:17.728196 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:17.728201 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:17.728499 (+ 298us) client_negotiation.cc:769] Sending connection context
0925 10:35:17.728705 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 10:35:17.729045 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":349,"threads_started":1}
I20260925 10:35:17.731892 13966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:35:17 dist-test-slave-dxdd krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260925 10:35:17.746429 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.732601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39577 (local address 127.0.0.1:53050)
0925 10:35:17.732777 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.732798 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:17.732894 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:17.733939 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:17.733943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:17.734435 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:17.734556 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:17.734563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:17.735833 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:17.735841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:17.736801 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:17.736812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:17.737570 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:17.737577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:17.737830 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:17.738416 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:17.738434 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:17.740265 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:17.743741 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:17.743746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:17.743750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:17.744138 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:17.744555 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:17.744560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:17.744564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:17.744692 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:17.745304 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:17.745314 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:17.745490 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 10:35:17.745646 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 10:35:17.745964 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":5}
I20260925 10:35:17.746462 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.732889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.0.97.61:39577)
0925 10:35:17.733560 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:17.733566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:17.733588 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:35:17.733626 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:17.733632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:17.733789 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:17.733945 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:17.734703 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:17.735690 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:17.736927 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:17.737437 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:17.737566 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:17.740461 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:17.740472 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:17.740483 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:17.740513 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:17.743583 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:17.744311 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:17.744320 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:17.744323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:17.744443 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:17.744800 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:17.744804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:17.744808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:17.745185 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:17.745442 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:17.745779 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 10:35:17.746031 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":525,"thread_start_us":392,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260925 10:35:17.749107 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39063
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36197,127.0.97.61:39577,127.0.97.60:39063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:35:17.749079 13966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:35:17.750128 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.749723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53534)
0925 10:35:17.749870 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.749985 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 10:35:17.750411 13916 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
I20260925 10:35:17.751942 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.751609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53538)
0925 10:35:17.751774 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.751840 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 10:35:17.752161 13966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
I20260925 10:35:17.801563 13966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } attempt: 1
I20260925 10:35:17.802819 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.802358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53544)
0925 10:35:17.802566 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.802665 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 10:35:17.803833 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.803475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53554)
0925 10:35:17.803653 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.803731 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 10:35:17.804056 13966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
I20260925 10:35:17.872262 13966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } attempt: 2
I20260925 10:35:17.873351 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.872935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53566)
0925 10:35:17.873116 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.873208 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 10:35:17.874244 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.873949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53570)
0925 10:35:17.874089 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.874145 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 10:35:17.874464 13966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
I20260925 10:35:17.936334 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } attempt: 5
Sep 25 10:35:17 dist-test-slave-dxdd krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260925 10:35:17.950821 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.937281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53052 (local address 127.0.97.61:39577)
0925 10:35:17.937461 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:17.937465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:17.937484 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:35:17.937602 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:17.937605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:17.937663 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:17.937744 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:17.938786 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:17.939477 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:17.941180 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:17.941594 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:17.941683 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:17.945422 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:17.945427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:17.945431 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:17.945460 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:17.948120 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:17.949008 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:17.949011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:17.949014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:17.949143 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:17.949593 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:17.949596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:17.949598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:17.949856 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:17.949960 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:17.950445 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 10:35:17.950599 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 10:35:17.950932 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.937074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39577 (local address 127.0.0.1:53052)
0925 10:35:17.937309 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.937333 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:17.937464 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:17.937823 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:17.937828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:17.938392 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:17.938621 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:17.938633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:17.939637 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:17.939643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:17.941044 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:17.941059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:17.941749 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:17.941756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:17.941967 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:17.942642 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:17.942688 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:17.945240 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:17.948280 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:17.948287 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:17.948294 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:17.948878 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:17.949266 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:17.949274 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:17.949279 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:17.949478 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:17.949974 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:17.949979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:17.950142 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 10:35:17.950324 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 10:35:17.950600 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 10:35:17.951958 13892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:35:17.952950 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.952523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53586)
0925 10:35:17.952695 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.952785 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 10:35:17.953887 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.953515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53598)
0925 10:35:17.953648 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.953761 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:35:17.954084 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
I20260925 10:35:17.965667 13966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } attempt: 3
I20260925 10:35:17.966701 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.966170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53602)
0925 10:35:17.966419 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.966511 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 10:35:17.967638 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:17.967247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53618)
0925 10:35:17.967460 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:17.967535 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260925 10:35:17.967840 13966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
I20260925 10:35:18.005295 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } attempt: 1
I20260925 10:35:18.006531 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.005913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53628)
0925 10:35:18.006140 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:18.006263 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 10:35:18.007570 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.007176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53644)
0925 10:35:18.007353 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:18.007440 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 10:35:18.007790 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
W20260925 10:35:18.070238 13973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:18.070537 13973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:18.070724 13973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 10:35:18.076016 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } attempt: 2
I20260925 10:35:18.077090 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.076634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53652)
0925 10:35:18.076838 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:18.076936 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 10:35:18.078055 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.077721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53654)
0925 10:35:18.077893 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:18.077958 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 10:35:18.078287 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
W20260925 10:35:18.078477 13973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:35:18.078596 13973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:18.078694 13973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:18.078773 13973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:35:18.078832 13973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:35:18.088847 13973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36197,127.0.97.61:39577,127.0.97.60:39063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:18.090051 13973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:18.091547 13973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:18.103214 13981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:18.103313 13978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:18.103245 13979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:18.104175 13973 server_base.cc:1034] running on GCE node
I20260925 10:35:18.104983 13973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:18.106349 13973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:18.107579 13973 hybrid_clock.cc:648] HybridClock initialized: now 1790332518107533 us; error 40 us; skew 500 ppm
I20260925 10:35:18.107921 13973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:18 dist-test-slave-dxdd krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:18.110888 13973 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260925 10:35:18.111083 13966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } attempt: 4
I20260925 10:35:18.112067 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.111632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53668)
0925 10:35:18.111848 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:18.111922 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 10:35:18.112476 13973 webserver.cc:460] Webserver started at http://127.0.97.60:34085/ using document root <none> and password file <none>
I20260925 10:35:18.113041 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.112755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53676)
0925 10:35:18.112885 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:18.112950 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 10:35:18.113189 13973 fs_manager.cc:359] Metadata directory not provided
W20260925 10:35:18.113219 13966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39063: connect: Connection refused (error 111)
I20260925 10:35:18.113337 13973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:18.113621 13973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:35:18.116458 13973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/data/instance:
uuid: "1247255f382844958c789401e6f0a52d"
format_stamp: "Formatted at 2026-09-25 10:35:18 on dist-test-slave-dxdd"
server_key: "f059ac6e5e0abcb8ef9fe950c24e1bd0"
server_key_iv: "4cac2943a2df14d684ea844d0ec7f5e3"
server_key_version: "encryptionkey@0"
I20260925 10:35:18.117254 13973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/wal/instance:
uuid: "1247255f382844958c789401e6f0a52d"
format_stamp: "Formatted at 2026-09-25 10:35:18 on dist-test-slave-dxdd"
server_key: "f059ac6e5e0abcb8ef9fe950c24e1bd0"
server_key_iv: "4cac2943a2df14d684ea844d0ec7f5e3"
server_key_version: "encryptionkey@0"
I20260925 10:35:18.122030 13973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 10:35:18.125243 13988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:18.126503 13973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:35:18.126683 13973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/wal
uuid: "1247255f382844958c789401e6f0a52d"
format_stamp: "Formatted at 2026-09-25 10:35:18 on dist-test-slave-dxdd"
server_key: "f059ac6e5e0abcb8ef9fe950c24e1bd0"
server_key_iv: "4cac2943a2df14d684ea844d0ec7f5e3"
server_key_version: "encryptionkey@0"
I20260925 10:35:18.126888 13973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:18.148025 13973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:18.152024 13973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:18.152302 13973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:18.170401 13973 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39063
I20260925 10:35:18.170446 14039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39063 every 8 connection(s)
I20260925 10:35:18.171691 13973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/data/info.pb
I20260925 10:35:18.175216 14040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:18.179647 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 13973
I20260925 10:35:18.179847 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332315635355-388-0/minicluster-data/master-2/wal/instance
I20260925 10:35:18.180370 388 external_mini_cluster.cc:1428] Setting key da73864474209692c5b5c37ae86431fa
I20260925 10:35:18.182503 14040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:35:18.184592 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } attempt: 3
Sep 25 10:35:18 dist-test-slave-dxdd krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 25 10:35:18 dist-test-slave-dxdd krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 25 10:35:18 dist-test-slave-dxdd krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:18.205029 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.185199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53684)
0925 10:35:18.185449 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:18.185469 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:18.185614 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:18.186910 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:18.186916 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:18.187493 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:18.187742 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:18.187756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.189719 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.189727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:18.191134 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:18.191158 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.192499 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.192505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:18.192714 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:18.194198 (+ 1484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:18.194220 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:18.197239 (+ 3019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:18.201848 (+ 4609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:18.201853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:18.201857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:18.202244 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:18.202660 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:18.202666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:18.202669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:18.202873 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:18.203614 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:18.203642 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:18.203838 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 10:35:18.204036 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 10:35:18.204319 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 10:35:18.205878 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.187291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.0.97.62:36197)
0925 10:35:18.187485 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:18.187491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:18.188269 (+ 778us) server_negotiation.cc:407] Connection header received
0925 10:35:18.188575 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:18.188581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:18.188651 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:18.188749 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:18.190157 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.191370 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.193366 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.193819 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.193954 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:18.198379 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:18.198389 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:18.198394 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:18.198434 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:18.201769 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.202742 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.202747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.202750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.202901 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.203405 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.203410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.203414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.203823 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:18.203964 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:18.204606 (+ 642us) server_negotiation.cc:299] Negotiation successful
0925 10:35:18.204808 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:35:18.205963 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.185317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.0.97.60:39063)
0925 10:35:18.186574 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:18.186580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:18.186604 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:35:18.186655 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:18.186661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:18.186745 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:18.186856 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:18.187952 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.189562 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.191323 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.192185 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.192359 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:18.197467 (+ 5108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:18.197479 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:18.197483 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:18.197535 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:18.201691 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.202391 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.202399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.202402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.202546 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.203003 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.203008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.203011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.203386 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:18.203553 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:18.204682 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0925 10:35:18.205259 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1190,"server-negotiator.queue_time_us":1095,"thread_start_us":935,"threads_started":1}
I20260925 10:35:18.205862 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.187356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36197 (local address 127.0.0.1:54048)
0925 10:35:18.188113 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:18.188172 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:18.188383 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:18.188840 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:18.188847 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:18.189644 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:18.190008 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:18.190021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.191537 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.191543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:18.193218 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:18.193254 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.194047 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.194054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:18.194251 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:18.194916 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:18.194947 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:18.198156 (+ 3209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:18.201914 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:18.201922 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:18.201926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:18.202580 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:18.203031 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:18.203034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:18.203036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:18.203269 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:18.204080 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:18.204087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:18.204285 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 10:35:18.204510 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 10:35:18.204815 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":3,"thread_start_us":380,"threads_started":1}
I20260925 10:35:18.206373 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.182189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.0.97.60:39063)
0925 10:35:18.182936 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:18.182947 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:18.182989 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:35:18.184532 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:18.184553 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:18.185083 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:18.185616 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:18.187393 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.189566 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.191417 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.192166 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.192357 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:18.197307 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:18.197322 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:18.197341 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:18.197379 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:18.200924 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.201664 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.201671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.201676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.201887 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.202412 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.202418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.202421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.203141 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:18.203487 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:18.203938 (+ 451us) server_negotiation.cc:299] Negotiation successful
0925 10:35:18.204339 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":554,"thread_start_us":329,"threads_started":1}
I20260925 10:35:18.207690 14040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:35:18.212632 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51: Bootstrap starting.
Sep 25 10:35:18 dist-test-slave-dxdd krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260925 10:35:18.215893 13892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:18.217301 13892 log.cc:824] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:18.220710 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51: No bootstrap required, opened a new log
I20260925 10:35:18.222406 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.208327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39577 (local address 127.0.0.1:53058)
0925 10:35:18.208792 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:18.208809 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:18.208933 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:18.209426 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:18.209430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:18.209771 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:18.209899 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:18.209907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.210944 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.210952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:18.211789 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:18.211800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.212507 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.212512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:18.212723 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:18.213323 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:18.213339 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:18.215340 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:18.219643 (+ 4303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:18.219648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:18.219652 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:18.220165 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:18.220616 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:18.220619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:18.220622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:18.220759 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:18.221437 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:18.221441 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:18.221588 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 10:35:18.221739 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 10:35:18.222017 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331}
I20260925 10:35:18.222429 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.208436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.0.97.61:39577)
0925 10:35:18.208634 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:18.208640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:18.208891 (+ 251us) server_negotiation.cc:407] Connection header received
0925 10:35:18.209046 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:18.209050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:18.209146 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:18.209251 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:18.210062 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.210811 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.211942 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.212365 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.212499 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:18.215550 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:18.215555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:18.215558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:18.215589 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:18.219486 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.220306 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.220311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.220315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.220479 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.220949 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.220955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.220961 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.221305 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:18.221470 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:18.221840 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 10:35:18.222044 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:35:18.223548 14040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:35:18 dist-test-slave-dxdd krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:18.231294 13892 raft_consensus.cc:348] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } }
I20260925 10:35:18.231691 13892 raft_consensus.cc:374] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 25 10:35:18 dist-test-slave-dxdd krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260925 10:35:18.231815 13892 raft_consensus.cc:729] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b569e12627ab4555acb04172ef255d51, State: Initialized, Role: FOLLOWER
I20260925 10:35:18.232523 13892 consensus_queue.cc:260] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } }
I20260925 10:35:18.233542 14047 sys_catalog.cc:455] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } } }
I20260925 10:35:18.234050 14047 sys_catalog.cc:458] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:35:18.234905 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.214923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.0.97.62:36197)
0925 10:35:18.215109 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:18.215115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:18.215134 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:35:18.215186 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:18.215191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:18.215265 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:18.215367 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:18.217199 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.218422 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.219799 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.220305 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.220440 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:18.226538 (+ 6098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:18.226544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:18.226550 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:18.226588 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:18.230546 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.231413 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.231420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.231424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.231567 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.232103 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.232111 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.232116 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.232479 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:18.232756 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:18.234273 (+ 1517us) server_negotiation.cc:299] Negotiation successful
0925 10:35:18.234534 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:35:18.234931 13892 sys_catalog.cc:564] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:35:18.245108 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.224300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53692)
0925 10:35:18.224488 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:18.224507 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:18.224601 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:18.224887 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:18.224891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:18.225261 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:18.225385 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:18.225416 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.226411 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.226418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:18.227730 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:18.227747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.228861 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.228866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:18.229098 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:18.229645 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:18.229667 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:18.232456 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:18.242415 (+ 9959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:18.242420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:18.242425 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:18.242929 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:18.243321 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:18.243325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:18.243328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:18.243477 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:18.244127 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:18.244131 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:18.244298 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 10:35:18.244424 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 10:35:18.244749 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 10:35:18.246417 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.224457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53692 (local address 127.0.97.60:39063)
0925 10:35:18.224598 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:18.224603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:18.224619 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:35:18.224708 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:18.224712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:18.224777 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:18.224871 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:18.225575 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.226253 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.227883 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.228543 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.228638 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:18.232618 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:18.232623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:18.232628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:18.232664 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:18.242261 (+ 9597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.243060 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.243065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.243068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.243209 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.243592 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.243596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.243598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.244001 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:18.245712 (+ 1711us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:18.245836 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 10:35:18.246051 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 10:35:18 dist-test-slave-dxdd krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260925 10:35:18.244845 13825 master_runner.cc:186] Error getting master registration for 127.0.97.61:39577: OK, instance_id { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" instance_seqno: 1790332517692458 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:35:18.255772 13825 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:35:18.257524 14040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d: Bootstrap starting.
W20260925 10:35:18.260535 14058 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:35:18.260764 14058 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:35:18.261493 14040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:18.262665 14040 log.cc:824] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:18.263559 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.242683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53072 (local address 127.0.97.61:39577)
0925 10:35:18.242852 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:18.242858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:18.242874 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:35:18.242926 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:18.242930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:18.243008 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:18.243098 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:18.244939 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.246416 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.248881 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.249840 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.249986 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:18.255926 (+ 5940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:18.255933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:18.255937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:18.255975 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:18.259601 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.260467 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.260472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.260475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.260676 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.261180 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.261186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.261190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.261527 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:18.261885 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:18.263009 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0925 10:35:18.263227 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:35:18.265789 14040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d: No bootstrap required, opened a new log
I20260925 10:35:18.268479 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:35:18.277741 14040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } }
I20260925 10:35:18.278262 14040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:18.278424 14040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1247255f382844958c789401e6f0a52d, State: Initialized, Role: FOLLOWER
I20260925 10:35:18.279217 14040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } }
I20260925 10:35:18.280146 14065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } } }
I20260925 10:35:18.280627 14065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:35:18.281826 14040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:35:18.304557 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.282474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.0.97.61:39577)
0925 10:35:18.282665 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:18.282674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:18.286693 (+ 4019us) server_negotiation.cc:407] Connection header received
0925 10:35:18.286945 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:18.286952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:18.287029 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:18.287110 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:18.289447 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.290577 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.292644 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.293336 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.293445 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:18.297223 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:18.297230 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:18.297236 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:18.297275 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:18.300859 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.301780 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.301786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.301789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.301942 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.302483 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.302489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.302492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.302872 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:18.303086 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:18.303929 (+ 843us) server_negotiation.cc:299] Negotiation successful
0925 10:35:18.304172 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 10:35:18.313886 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.280993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.0.97.62:36197)
0925 10:35:18.281212 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:18.281221 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:18.286294 (+ 5073us) server_negotiation.cc:407] Connection header received
0925 10:35:18.287636 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:18.287645 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:18.287771 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:18.287855 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:18.293053 (+ 5198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.294187 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.300298 (+ 6111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.301006 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.301095 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:18.306615 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:18.306622 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:18.306627 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:18.306665 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:18.310317 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.311303 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.311309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.311312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.311482 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.311937 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.311944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.311947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.312305 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:18.312602 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:18.313165 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 10:35:18.313412 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 10:35:18.318703 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.286815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53704 (local address 127.0.97.60:39063)
0925 10:35:18.287024 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:18.287030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:18.300686 (+ 13656us) server_negotiation.cc:407] Connection header received
0925 10:35:18.301077 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:18.301083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:18.301160 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:18.301243 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:18.304264 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.305573 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.307776 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.308515 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.308630 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:18.311816 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:18.311822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:18.311826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:18.311866 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:18.315259 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.316135 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.316140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.316146 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.316312 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.316739 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.316745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.316749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.317127 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:18.317462 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:18.318089 (+ 627us) server_negotiation.cc:299] Negotiation successful
0925 10:35:18.318262 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 10:35:18.327216 14079 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:35:18.327335 14079 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:35:18.328761 13973 master_runner.cc:186] Error getting master registration for 127.0.97.61:39577: OK, instance_id { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" instance_seqno: 1790332517692458 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:35:18.336536 13973 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:35:18.337003 14047 raft_consensus.cc:482] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:35:18.337283 14047 raft_consensus.cc:504] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } }
I20260925 10:35:18.338798 14047 leader_election.cc:288] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2229f7236e504be9bb6d48dbf732883a (127.0.97.61:39577), 1247255f382844958c789401e6f0a52d (127.0.97.60:39063)
I20260925 10:35:18.339691 14015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b569e12627ab4555acb04172ef255d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1247255f382844958c789401e6f0a52d" is_pre_election: true
I20260925 10:35:18.340340 14015 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b569e12627ab4555acb04172ef255d51 in term 0.
I20260925 10:35:18.340961 13842 leader_election.cc:302] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1247255f382844958c789401e6f0a52d, b569e12627ab4555acb04172ef255d51; no voters:
I20260925 10:35:18.341363 14047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:35:18.341535 14047 raft_consensus.cc:482] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:35:18.341652 14047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:18.345018 14047 raft_consensus.cc:504] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } }
I20260925 10:35:18.347530 14047 leader_election.cc:288] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [CANDIDATE]: Term 1 election: Requested vote from peers 2229f7236e504be9bb6d48dbf732883a (127.0.97.61:39577), 1247255f382844958c789401e6f0a52d (127.0.97.60:39063)
I20260925 10:35:18.347628 14015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b569e12627ab4555acb04172ef255d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1247255f382844958c789401e6f0a52d"
I20260925 10:35:18.347836 14015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:18.351207 14015 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b569e12627ab4555acb04172ef255d51 in term 1.
I20260925 10:35:18.351629 13842 leader_election.cc:302] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1247255f382844958c789401e6f0a52d, b569e12627ab4555acb04172ef255d51; no voters:
I20260925 10:35:18.352010 14047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:35:18.354482 14047 raft_consensus.cc:686] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [term 1 LEADER]: Becoming Leader. State: Replica: b569e12627ab4555acb04172ef255d51, State: Running, Role: LEADER
I20260925 10:35:18.355067 14047 consensus_queue.cc:237] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } }
I20260925 10:35:18.358109 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b569e12627ab4555acb04172ef255d51. Latest consensus state: current_term: 1 leader_uuid: "b569e12627ab4555acb04172ef255d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } } }
I20260925 10:35:18.358554 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:18.372189 14083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:35:18.376520 14083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:35:18.391666 14015 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d [term 1 FOLLOWER]: Refusing update from remote peer b569e12627ab4555acb04172ef255d51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:35:18.392730 14081 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:35:18.404249 13994 tablet.cc:2369] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:35:18.411353 14065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b569e12627ab4555acb04172ef255d51. Latest consensus state: current_term: 1 leader_uuid: "b569e12627ab4555acb04172ef255d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } } }
I20260925 10:35:18.411654 14065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:35:18.412691 14084 mvcc.cc:205] Tried to move back new op lower bound from 7333201995322839040 to 7333201995187503104. Current Snapshot: MvccSnapshot[applied={T|T < 7333201995322839040}]
I20260925 10:35:18.413054 13966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } attempt: 5
I20260925 10:35:18.423401 14047 sys_catalog.cc:455] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b569e12627ab4555acb04172ef255d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } } }
I20260925 10:35:18.423723 14047 sys_catalog.cc:458] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:18.424572 14083 catalog_manager.cc:1347] Generated new cluster ID: 356507dc19d44c1591e2e7ca0eac1c8c
I20260925 10:35:18.424669 14083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:35:18 dist-test-slave-dxdd krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260925 10:35:18.448684 14085 mvcc.cc:205] Tried to move back new op lower bound from 7333201995322839040 to 7333201995187503104. Current Snapshot: MvccSnapshot[applied={T|T < 7333201995322839040}]
I20260925 10:35:18.455353 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.414726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.0.97.60:39063)
0925 10:35:18.414886 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:18.414893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:18.415522 (+ 629us) server_negotiation.cc:407] Connection header received
0925 10:35:18.415580 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:18.415585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:18.415687 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:18.415772 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:18.416953 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.418131 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.420236 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.420913 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.421036 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:18.436559 (+ 15523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:18.436566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:18.436572 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:18.436612 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:18.442943 (+ 6331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.446527 (+ 3584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.446574 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.446578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.447064 (+ 486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:18.449630 (+ 2566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:18.449637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:18.449641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:18.450482 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:18.450769 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:18.453919 (+ 3150us) server_negotiation.cc:299] Negotiation successful
0925 10:35:18.454485 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:35:18.461316 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:18.414723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39063 (local address 127.0.0.1:53714)
0925 10:35:18.415271 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:18.415290 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:18.415400 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:18.415981 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:18.415987 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:18.416495 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:18.416705 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:18.416718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.418367 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.418373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:18.419839 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:18.419855 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:18.421281 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:18.421287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:18.421484 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:18.422276 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:18.422344 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:18.435909 (+ 13565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:18.443232 (+ 7323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:18.443238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:18.443275 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:18.446054 (+ 2779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:18.447627 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:18.447633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:18.447636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:18.448906 (+ 1270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:18.451250 (+ 2344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:18.451258 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:18.452579 (+ 1321us) client_negotiation.cc:769] Sending connection context
0925 10:35:18.455840 (+ 3261us) client_negotiation.cc:240] Negotiation successful
0925 10:35:18.457689 (+ 1849us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429}
I20260925 10:35:18.466473 14065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b569e12627ab4555acb04172ef255d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } } }
I20260925 10:35:18.466746 14065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:35:18.468361 13966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a: Bootstrap starting.
I20260925 10:35:18.472548 13966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:18.474071 14083 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:35:18.477228 14083 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:35:18.473800 13966 log.cc:824] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:18.497900 14083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51: Generated new TSK 0
I20260925 10:35:18.499212 13966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a: No bootstrap required, opened a new log
I20260925 10:35:18.500128 14083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:35:18.515507 13966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } }
I20260925 10:35:18.515919 13966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:18.516036 13966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2229f7236e504be9bb6d48dbf732883a, State: Initialized, Role: FOLLOWER
I20260925 10:35:18.516772 13966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } }
I20260925 10:35:18.526852 14091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } } }
I20260925 10:35:18.528081 13966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:35:18.528415 13939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:18.534938 14091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:35:18.528587 13941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b569e12627ab4555acb04172ef255d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2229f7236e504be9bb6d48dbf732883a" is_pre_election: true
I20260925 10:35:18.528883 13940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b569e12627ab4555acb04172ef255d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2229f7236e504be9bb6d48dbf732883a"
I20260925 10:35:18.539963 13939 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a [term 1 FOLLOWER]: Refusing update from remote peer b569e12627ab4555acb04172ef255d51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:35:18.540740 14047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:35:18.583464 14091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b569e12627ab4555acb04172ef255d51. Latest consensus state: current_term: 1 leader_uuid: "b569e12627ab4555acb04172ef255d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } } }
I20260925 10:35:18.583817 14091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:35:18.584278 14099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b569e12627ab4555acb04172ef255d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } } }
I20260925 10:35:18.584753 14099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:35:18.612920 14111 catalog_manager.cc:1259] Loaded cluster ID: 356507dc19d44c1591e2e7ca0eac1c8c
I20260925 10:35:18.613016 14111 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a: loading cluster ID for follower catalog manager: success
I20260925 10:35:18.616609 14111 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a: acquiring CA information for follower catalog manager: success
I20260925 10:35:18.618968 14111 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2229f7236e504be9bb6d48dbf732883a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:35:19.071148 14047 sys_catalog.cc:455] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b569e12627ab4555acb04172ef255d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b569e12627ab4555acb04172ef255d51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36197 } } peers { permanent_uuid: "2229f7236e504be9bb6d48dbf732883a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39577 } } peers { permanent_uuid: "1247255f382844958c789401e6f0a52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39063 } } }
I20260925 10:35:19.071473 14047 sys_catalog.cc:458] T 00000000000000000000000000000000 P b569e12627ab4555acb04172ef255d51 [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:19.329131 14079 catalog_manager.cc:1259] Loaded cluster ID: 356507dc19d44c1591e2e7ca0eac1c8c
I20260925 10:35:19.329275 14079 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d: loading cluster ID for follower catalog manager: success
I20260925 10:35:19.331915 14079 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d: acquiring CA information for follower catalog manager: success
I20260925 10:35:19.333618 14079 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1247255f382844958c789401e6f0a52d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:35:19.643239 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 13825
I20260925 10:35:19.644416 13887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:19.759014 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 13825
I20260925 10:35:19.779451 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 13899
I20260925 10:35:19.780479 13961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:19.885392 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 13899
I20260925 10:35:19.905660 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 13973
I20260925 10:35:19.906817 14035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:20.010561 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 13973
2026-09-25T10:35:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4488 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:35:20 dist-test-slave-dxdd krb5kdc[14121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:35:20 dist-test-slave-dxdd krb5kdc[14121](info): set up 2 sockets
Sep 25 10:35:20 dist-test-slave-dxdd krb5kdc[14121](info): commencing operation
krb5kdc: starting...
W20260925 10:35:21.090566 388 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:35:21 dist-test-slave-dxdd krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:35:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:35:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:35:21.305946 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39971
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:35:21.613802 14134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:21.614135 14134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:21.614321 14134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:21.621366 14134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:35:21.621515 14134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:21.621626 14134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:21.621730 14134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:35:21.621819 14134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:35:21.631657 14134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:21.632896 14134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:21.634150 14134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:21.645602 14139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:21.645655 14140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:21.645566 14142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:21.646997 14134 server_base.cc:1034] running on GCE node
I20260925 10:35:21.647660 14134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:21.648923 14134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:21.650161 14134 hybrid_clock.cc:648] HybridClock initialized: now 1790332521650115 us; error 50 us; skew 500 ppm
I20260925 10:35:21.650552 14134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:21 dist-test-slave-dxdd krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:21.653518 14134 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:35:21.655045 14134 webserver.cc:460] Webserver started at http://127.0.97.62:41973/ using document root <none> and password file <none>
I20260925 10:35:21.655719 14134 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:21.655864 14134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:21.656175 14134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:35:21.659071 14134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "02b3e713d0884172915dfe761701009e"
format_stamp: "Formatted at 2026-09-25 10:35:21 on dist-test-slave-dxdd"
server_key: "548424e9724e85660e1c8537b1ea3996"
server_key_iv: "e2d751441b4fc1312964e384af3f6e82"
server_key_version: "encryptionkey@0"
I20260925 10:35:21.659781 14134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "02b3e713d0884172915dfe761701009e"
format_stamp: "Formatted at 2026-09-25 10:35:21 on dist-test-slave-dxdd"
server_key: "548424e9724e85660e1c8537b1ea3996"
server_key_iv: "e2d751441b4fc1312964e384af3f6e82"
server_key_version: "encryptionkey@0"
I20260925 10:35:21.664707 14134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 10:35:21.667906 14149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:21.669085 14134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:35:21.669260 14134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "02b3e713d0884172915dfe761701009e"
format_stamp: "Formatted at 2026-09-25 10:35:21 on dist-test-slave-dxdd"
server_key: "548424e9724e85660e1c8537b1ea3996"
server_key_iv: "e2d751441b4fc1312964e384af3f6e82"
server_key_version: "encryptionkey@0"
I20260925 10:35:21.669442 14134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:21.688562 14134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:21.692251 14134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:21.692521 14134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:21.710621 14134 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39971
I20260925 10:35:21.710652 14200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39971 every 8 connection(s)
I20260925 10:35:21.711859 14134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:35:21.713581 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 14134
I20260925 10:35:21.713794 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:35:21.714637 388 external_mini_cluster.cc:1428] Setting key 7eae0ec35864af4c2436af1d9bc013bc
I20260925 10:35:21.716246 14201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 10:35:21 dist-test-slave-dxdd krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:21.729759 14201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e: Bootstrap starting.
I20260925 10:35:21.733060 14201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:21.734021 14201 log.cc:824] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:21.736706 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:21.716489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.0.97.62:39971)
0925 10:35:21.717226 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:21.717237 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:21.717276 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:35:21.718898 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:21.718917 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:21.719274 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:21.719583 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:21.721343 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:21.722761 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:21.724570 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:21.725181 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:21.725307 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:21.729899 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:21.729910 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:21.729929 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:21.729970 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:21.732877 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:21.733609 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:21.733616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:21.733621 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:21.733770 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:21.734152 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:21.734159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:21.734162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:21.734713 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:21.735030 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:21.735444 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 10:35:21.735843 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":357,"threads_started":1}
I20260925 10:35:21.736804 14201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e: No bootstrap required, opened a new log
I20260925 10:35:21.740481 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:35:21.744582 14201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b3e713d0884172915dfe761701009e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39971 } }
I20260925 10:35:21.744930 14201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:21.745031 14201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b3e713d0884172915dfe761701009e, State: Initialized, Role: FOLLOWER
I20260925 10:35:21.745539 14201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b3e713d0884172915dfe761701009e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39971 } }
I20260925 10:35:21.745774 14201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:35:21.745914 14201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:35:21.746081 14201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:21.748154 14201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b3e713d0884172915dfe761701009e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39971 } }
I20260925 10:35:21.748611 14201 leader_election.cc:302] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02b3e713d0884172915dfe761701009e; no voters:
I20260925 10:35:21.749176 14201 leader_election.cc:288] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:35:21.749312 14206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:35:21.749972 14206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [term 1 LEADER]: Becoming Leader. State: Replica: 02b3e713d0884172915dfe761701009e, State: Running, Role: LEADER
I20260925 10:35:21.750559 14206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b3e713d0884172915dfe761701009e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39971 } }
I20260925 10:35:21.751240 14201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:35:21.753058 14207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02b3e713d0884172915dfe761701009e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b3e713d0884172915dfe761701009e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39971 } } }
I20260925 10:35:21.753484 14207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:21.758569 14208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02b3e713d0884172915dfe761701009e. Latest consensus state: current_term: 1 leader_uuid: "02b3e713d0884172915dfe761701009e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b3e713d0884172915dfe761701009e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39971 } } }
I20260925 10:35:21.758993 14208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:21.759289 14215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:35:21.764258 14215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:35:21.772308 14215 catalog_manager.cc:1347] Generated new cluster ID: 68e50601d8d54a7bb4a855c98d7fca6a
I20260925 10:35:21.772413 14215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:35:21.786837 14215 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:35:21.788527 14215 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:35:21.800420 14215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02b3e713d0884172915dfe761701009e: Generated new TSK 0
I20260925 10:35:21.801194 14215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:35:21.852708 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 14134
I20260925 10:35:21.853928 14196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:21.946733 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 14134
2026-09-25T10:35:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1932 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:35:22 dist-test-slave-dxdd krb5kdc[14229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:35:22 dist-test-slave-dxdd krb5kdc[14229](info): set up 2 sockets
Sep 25 10:35:22 dist-test-slave-dxdd krb5kdc[14229](info): commencing operation
krb5kdc: starting...
W20260925 10:35:23.030709 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.008s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:35:23 dist-test-slave-dxdd krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:35:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:35:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:35:23.265605 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43783
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:35:23.576737 14242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:23.577032 14242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:23.577181 14242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:23.583987 14242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:35:23.584128 14242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:23.584206 14242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:23.584334 14242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:35:23.584399 14242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:35:23.593390 14242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:23.594631 14242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:23.595932 14242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:23.607460 14248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:23.607540 14247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:23.607506 14250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:23.608440 14242 server_base.cc:1034] running on GCE node
I20260925 10:35:23.609310 14242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:23.610822 14242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:23.612048 14242 hybrid_clock.cc:648] HybridClock initialized: now 1790332523611959 us; error 76 us; skew 500 ppm
I20260925 10:35:23.612388 14242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:23 dist-test-slave-dxdd krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:23.615506 14242 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:35:23.616873 14242 webserver.cc:460] Webserver started at http://127.0.97.62:36329/ using document root <none> and password file <none>
I20260925 10:35:23.617493 14242 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:23.617590 14242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:23.617833 14242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:35:23.620725 14242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "6b0d39c466504eef8ec3e7d55023612f"
format_stamp: "Formatted at 2026-09-25 10:35:23 on dist-test-slave-dxdd"
server_key: "b8c18ba86bea496b51b7e392d73a436d"
server_key_iv: "99e6e78019097f25ff7a19bf803323ee"
server_key_version: "encryptionkey@0"
I20260925 10:35:23.621392 14242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "6b0d39c466504eef8ec3e7d55023612f"
format_stamp: "Formatted at 2026-09-25 10:35:23 on dist-test-slave-dxdd"
server_key: "b8c18ba86bea496b51b7e392d73a436d"
server_key_iv: "99e6e78019097f25ff7a19bf803323ee"
server_key_version: "encryptionkey@0"
I20260925 10:35:23.626155 14242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 10:35:23.629340 14257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:23.630625 14242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:35:23.630743 14242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "6b0d39c466504eef8ec3e7d55023612f"
format_stamp: "Formatted at 2026-09-25 10:35:23 on dist-test-slave-dxdd"
server_key: "b8c18ba86bea496b51b7e392d73a436d"
server_key_iv: "99e6e78019097f25ff7a19bf803323ee"
server_key_version: "encryptionkey@0"
I20260925 10:35:23.630880 14242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:23.641636 14242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:23.645711 14242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:23.645929 14242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:23.663887 14242 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43783
I20260925 10:35:23.663975 14308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43783 every 8 connection(s)
I20260925 10:35:23.665242 14242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:35:23.668802 14309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:23.675177 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 14242
I20260925 10:35:23.675381 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:35:23.676184 388 external_mini_cluster.cc:1428] Setting key 92eba18241c063417b9dc9b8fd106947
I20260925 10:35:23.679733 14309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f: Bootstrap starting.
I20260925 10:35:23.683157 14309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:23.684190 14309 log.cc:824] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:23.687047 14309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f: No bootstrap required, opened a new log
Sep 25 10:35:23 dist-test-slave-dxdd krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:23.697587 14309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } }
I20260925 10:35:23.698011 14309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:23.698124 14309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0d39c466504eef8ec3e7d55023612f, State: Initialized, Role: FOLLOWER
I20260925 10:35:23.698781 14309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } }
I20260925 10:35:23.699057 14309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:35:23.699206 14309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:35:23.699398 14309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:23.700181 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:23.678154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55804 (local address 127.0.97.62:43783)
0925 10:35:23.678959 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:23.678971 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:23.679010 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:35:23.680686 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:23.680707 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:23.681090 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:23.681393 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:23.683189 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:23.684539 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:23.686370 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:23.687004 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:23.687189 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:23.691724 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:23.691735 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:23.691756 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:23.691817 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:23.695775 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:23.696432 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:23.696441 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:23.696449 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:23.696662 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:23.697012 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:23.697016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:23.697019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:23.697677 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:23.697964 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:23.698348 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 10:35:23.698741 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":419,"threads_started":1}
I20260925 10:35:23.701928 14309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } }
I20260925 10:35:23.702533 14309 leader_election.cc:302] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b0d39c466504eef8ec3e7d55023612f; no voters:
I20260925 10:35:23.703159 14309 leader_election.cc:288] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:35:23.703300 14314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:35:23.704087 14314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 1 LEADER]: Becoming Leader. State: Replica: 6b0d39c466504eef8ec3e7d55023612f, State: Running, Role: LEADER
I20260925 10:35:23.704691 14314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } }
I20260925 10:35:23.705323 14309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:35:23.707657 14316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b0d39c466504eef8ec3e7d55023612f. Latest consensus state: current_term: 1 leader_uuid: "6b0d39c466504eef8ec3e7d55023612f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } } }
I20260925 10:35:23.708024 14316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:23.707752 14315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b0d39c466504eef8ec3e7d55023612f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } } }
I20260925 10:35:23.708163 14315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:23.709916 14320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:35:23.715893 14320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:35:23.723892 14320 catalog_manager.cc:1347] Generated new cluster ID: bb3956981dec43f3bb31901ee892836f
I20260925 10:35:23.724006 14320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:35:23.738726 14320 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:35:23.740439 14320 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:35:23.746260 14320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f: Generated new TSK 0
I20260925 10:35:23.747140 14320 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:35:23.798156 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43783
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:35:24.115521 14335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:24.115787 14335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:24.116005 14335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:24.123173 14335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:24.123277 14335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:24.123386 14335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:35:24.132792 14335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:24.134088 14335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:24.135421 14335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:24.150842 14341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:24.150884 14343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:24.151067 14340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:24.151562 14335 server_base.cc:1034] running on GCE node
I20260925 10:35:24.152204 14335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:24.153082 14335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:24.154356 14335 hybrid_clock.cc:648] HybridClock initialized: now 1790332524154277 us; error 53 us; skew 500 ppm
I20260925 10:35:24.154690 14335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:24 dist-test-slave-dxdd krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:24.157728 14335 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:35:24.159200 14335 webserver.cc:460] Webserver started at http://127.0.97.1:34191/ using document root <none> and password file <none>
I20260925 10:35:24.159863 14335 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:24.160006 14335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:24.160319 14335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:35:24.163164 14335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed"
format_stamp: "Formatted at 2026-09-25 10:35:24 on dist-test-slave-dxdd"
server_key: "742019e08499c25a682dfa77ef966d88"
server_key_iv: "5df8b1cc2fd5197bdc21cd60a5ab5f20"
server_key_version: "encryptionkey@0"
I20260925 10:35:24.163880 14335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed"
format_stamp: "Formatted at 2026-09-25 10:35:24 on dist-test-slave-dxdd"
server_key: "742019e08499c25a682dfa77ef966d88"
server_key_iv: "5df8b1cc2fd5197bdc21cd60a5ab5f20"
server_key_version: "encryptionkey@0"
I20260925 10:35:24.168867 14335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 10:35:24.172264 14350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:24.173497 14335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:35:24.173705 14335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed"
format_stamp: "Formatted at 2026-09-25 10:35:24 on dist-test-slave-dxdd"
server_key: "742019e08499c25a682dfa77ef966d88"
server_key_iv: "5df8b1cc2fd5197bdc21cd60a5ab5f20"
server_key_version: "encryptionkey@0"
I20260925 10:35:24.173899 14335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:24.192651 14335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:24.196784 14335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:24.197095 14335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:24.198087 14335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:24.199602 14335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:24.199704 14335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:24.199856 14335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:24.199930 14335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:24.232678 14335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33633
I20260925 10:35:24.232722 14462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33633 every 8 connection(s)
I20260925 10:35:24.233942 14335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:35:24.238354 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 14335
I20260925 10:35:24.238565 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:35:24.239352 388 external_mini_cluster.cc:1428] Setting key 5e0a33caaeb3e8704207d05dc5bc47a2
Sep 25 10:35:24 dist-test-slave-dxdd krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:24.255362 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:24.236455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53141 (local address 127.0.97.62:43783)
0925 10:35:24.236913 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:24.236917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:24.237673 (+ 756us) server_negotiation.cc:407] Connection header received
0925 10:35:24.240240 (+ 2567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:24.240246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:24.240339 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:24.240462 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:24.242133 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:24.242905 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:24.245338 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:24.245851 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:24.245990 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:24.249617 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:24.249623 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:24.249628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:24.249679 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:24.252167 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:24.253088 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:24.253092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:24.253094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:24.253210 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:24.253665 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:24.253668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:24.253670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:24.253935 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:24.254040 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:24.254947 (+ 907us) server_negotiation.cc:299] Negotiation successful
0925 10:35:24.255106 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":256,"threads_started":1}
I20260925 10:35:24.256892 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:24.236849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:53141)
0925 10:35:24.237519 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:24.237554 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:24.239963 (+ 2409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:24.240697 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:24.240705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:24.241199 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:24.241957 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:24.241980 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:24.243051 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:24.243059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:24.245196 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:24.245214 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:24.246072 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:24.246079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:24.246367 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:24.247069 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:24.247093 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:24.249401 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:24.252317 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:24.252328 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:24.252346 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:24.252973 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:24.253313 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:24.253320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:24.253327 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:24.253562 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:24.254056 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:24.254066 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:24.254558 (+ 492us) client_negotiation.cc:769] Sending connection context
0925 10:35:24.254882 (+ 324us) client_negotiation.cc:240] Negotiation successful
0925 10:35:24.255446 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":266,"threads_started":1}
I20260925 10:35:24.258492 14463 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43783
I20260925 10:35:24.258800 14463 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:24.259563 14463 heartbeater.cc:511] Master 127.0.97.62:43783 requested a full tablet report, sending...
I20260925 10:35:24.261508 14274 ts_manager.cc:194] Registered new tserver with Master: dc3d44bdb2eb4acabbe96820ce4db1ed (127.0.97.1:33633)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260925 10:35:24.262962 14274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:35:24.289332 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43783
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:35:24.597913 14469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:24.598160 14469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:24.598325 14469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:24.605309 14469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:24.605439 14469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:24.605605 14469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:35:24.614655 14469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:24.615736 14469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:24.616979 14469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:24.629701 14475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:24.629660 14477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:24.629752 14474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:24.630514 14469 server_base.cc:1034] running on GCE node
I20260925 10:35:24.631134 14469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:24.632004 14469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:24.633226 14469 hybrid_clock.cc:648] HybridClock initialized: now 1790332524633190 us; error 49 us; skew 500 ppm
I20260925 10:35:24.633579 14469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:24 dist-test-slave-dxdd krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:24.636674 14469 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:35:24.638100 14469 webserver.cc:460] Webserver started at http://127.0.97.2:36013/ using document root <none> and password file <none>
I20260925 10:35:24.638803 14469 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:24.638950 14469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:24.639240 14469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:35:24.641973 14469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "7e146c068db545d596246e5e6cbf7fd6"
format_stamp: "Formatted at 2026-09-25 10:35:24 on dist-test-slave-dxdd"
server_key: "4aadb17a7a3e107a604fdcf784b6e10e"
server_key_iv: "a9364526c51d3b8c1263d31b37d132ec"
server_key_version: "encryptionkey@0"
I20260925 10:35:24.642735 14469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7e146c068db545d596246e5e6cbf7fd6"
format_stamp: "Formatted at 2026-09-25 10:35:24 on dist-test-slave-dxdd"
server_key: "4aadb17a7a3e107a604fdcf784b6e10e"
server_key_iv: "a9364526c51d3b8c1263d31b37d132ec"
server_key_version: "encryptionkey@0"
I20260925 10:35:24.647528 14469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 10:35:24.650911 14484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:24.652141 14469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:35:24.652314 14469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "7e146c068db545d596246e5e6cbf7fd6"
format_stamp: "Formatted at 2026-09-25 10:35:24 on dist-test-slave-dxdd"
server_key: "4aadb17a7a3e107a604fdcf784b6e10e"
server_key_iv: "a9364526c51d3b8c1263d31b37d132ec"
server_key_version: "encryptionkey@0"
I20260925 10:35:24.652501 14469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:24.677476 14469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:24.681736 14469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:24.682030 14469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:24.682993 14469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:24.684230 14469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:24.684340 14469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:24.684489 14469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:24.684561 14469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:24.716284 14469 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37349
I20260925 10:35:24.716367 14596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37349 every 8 connection(s)
I20260925 10:35:24.717553 14469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:35:24.719492 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 14469
I20260925 10:35:24.719683 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:35:24.720148 388 external_mini_cluster.cc:1428] Setting key 60879b5050143a504a65f6ddae9ccb24
Sep 25 10:35:24 dist-test-slave-dxdd krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:24.738164 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:24.720299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41795 (local address 127.0.97.62:43783)
0925 10:35:24.720518 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:24.720524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:24.721826 (+ 1302us) server_negotiation.cc:407] Connection header received
0925 10:35:24.723722 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:24.723730 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:24.723828 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:24.723940 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:24.725324 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:24.726573 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:24.728031 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:24.728749 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:24.728849 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:24.732610 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:24.732616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:24.732619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:24.732650 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:24.735275 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:24.735975 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:24.735982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:24.735991 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:24.736114 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:24.736530 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:24.736536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:24.736540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:24.736793 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:24.736966 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:24.737591 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 10:35:24.737812 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:35:24.739107 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:24.720862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:41795)
0925 10:35:24.721667 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:24.721721 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:24.723466 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:24.724121 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:24.724129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:24.724623 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:24.725152 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:24.725169 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:24.726699 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:24.726703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:24.727889 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:24.727900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:24.728983 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:24.728991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:24.729302 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:24.730032 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:24.730053 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:24.732397 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:24.735436 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:24.735444 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:24.735458 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:24.735848 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:24.736221 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:24.736227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:24.736230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:24.736417 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:24.736927 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:24.736934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:24.737242 (+ 308us) client_negotiation.cc:769] Sending connection context
0925 10:35:24.737505 (+ 263us) client_negotiation.cc:240] Negotiation successful
0925 10:35:24.737998 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":346,"threads_started":1}
I20260925 10:35:24.740651 14597 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43783
I20260925 10:35:24.740939 14597 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:24.741601 14597 heartbeater.cc:511] Master 127.0.97.62:43783 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260925 10:35:24.743005 14274 ts_manager.cc:194] Registered new tserver with Master: 7e146c068db545d596246e5e6cbf7fd6 (127.0.97.2:37349)
I20260925 10:35:24.743952 14274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:35:24.769631 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43783
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:35:25.085817 14602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:25.086072 14602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:25.086163 14602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:25.093087 14602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:25.093179 14602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:25.093292 14602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:35:25.102478 14602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:25.103552 14602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:25.104759 14602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:25.117311 14608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:25.117276 14610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:25.117354 14607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:25.118132 14602 server_base.cc:1034] running on GCE node
I20260925 10:35:25.118729 14602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:25.119627 14602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:25.120858 14602 hybrid_clock.cc:648] HybridClock initialized: now 1790332525120820 us; error 52 us; skew 500 ppm
I20260925 10:35:25.121165 14602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:25 dist-test-slave-dxdd krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:25.124157 14602 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:35:25.125504 14602 webserver.cc:460] Webserver started at http://127.0.97.3:39733/ using document root <none> and password file <none>
I20260925 10:35:25.126120 14602 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:25.126217 14602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:25.126531 14602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:35:25.129142 14602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1"
format_stamp: "Formatted at 2026-09-25 10:35:25 on dist-test-slave-dxdd"
server_key: "95e9a4c48d4dec08c6b290347a2e710e"
server_key_iv: "7520057a35ce30d2dd607b0ef929c575"
server_key_version: "encryptionkey@0"
I20260925 10:35:25.129806 14602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1"
format_stamp: "Formatted at 2026-09-25 10:35:25 on dist-test-slave-dxdd"
server_key: "95e9a4c48d4dec08c6b290347a2e710e"
server_key_iv: "7520057a35ce30d2dd607b0ef929c575"
server_key_version: "encryptionkey@0"
I20260925 10:35:25.134567 14602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 10:35:25.137816 14617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:25.139290 14602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:35:25.139457 14602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1"
format_stamp: "Formatted at 2026-09-25 10:35:25 on dist-test-slave-dxdd"
server_key: "95e9a4c48d4dec08c6b290347a2e710e"
server_key_iv: "7520057a35ce30d2dd607b0ef929c575"
server_key_version: "encryptionkey@0"
I20260925 10:35:25.139637 14602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:25.171324 14602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:25.175230 14602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:25.175532 14602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:25.176477 14602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:25.177915 14602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:25.177997 14602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:25.178184 14602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:25.178236 14602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:25.211114 14602 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40949
I20260925 10:35:25.211163 14729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40949 every 8 connection(s)
I20260925 10:35:25.212486 14602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:35:25.219800 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 14602
I20260925 10:35:25.220005 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:35:25.220456 388 external_mini_cluster.cc:1428] Setting key bfc38eeea767c622ec98ba1e50045b24
Sep 25 10:35:25 dist-test-slave-dxdd krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:25.231686 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.214953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54843 (local address 127.0.97.62:43783)
0925 10:35:25.215134 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:25.215139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:25.216320 (+ 1181us) server_negotiation.cc:407] Connection header received
0925 10:35:25.217991 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:25.217996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:25.218058 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:25.218147 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:25.219755 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.220663 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.222768 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.223263 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.223394 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.226711 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:25.226715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:25.226718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:25.226751 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:25.229218 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:25.229842 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:25.229846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:25.229848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:25.229958 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:25.230354 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:25.230364 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:25.230367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:25.230632 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:25.230737 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:25.231306 (+ 569us) server_negotiation.cc:299] Negotiation successful
0925 10:35:25.231453 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:35:25.232584 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.215462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:54843)
0925 10:35:25.216153 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:25.216196 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:25.217748 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:25.218394 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:25.218402 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:25.218923 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:25.219576 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:25.219593 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.220818 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.220822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:25.222634 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:25.222657 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.223462 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.223467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:25.223658 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.224386 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:25.224412 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:25.226531 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:25.229345 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:25.229351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:25.229361 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:25.229740 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:25.230043 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:25.230046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:25.230048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:25.230217 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:25.230733 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:25.230738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:25.231002 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 10:35:25.231217 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 10:35:25.231546 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":290,"threads_started":1}
I20260925 10:35:25.234016 14730 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43783
I20260925 10:35:25.234370 14730 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:25.235195 14730 heartbeater.cc:511] Master 127.0.97.62:43783 requested a full tablet report, sending...
I20260925 10:35:25.236724 14274 ts_manager.cc:194] Registered new tserver with Master: 6f4d65d8f3ef48eeae930d5b0531ffa1 (127.0.97.3:40949)
I20260925 10:35:25.237502 14274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54843
I20260925 10:35:25.247519 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:35:25.263157 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.251663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.0.97.62:43783)
0925 10:35:25.251897 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:25.251903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:25.252172 (+ 269us) server_negotiation.cc:407] Connection header received
0925 10:35:25.252294 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:25.252298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:25.252356 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:25.252427 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:25.253629 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.254371 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.255649 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.256033 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.256144 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.258367 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:25.258372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:25.258375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:25.258402 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:25.260877 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:25.261513 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:25.261517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:25.261519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:25.261632 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:25.261924 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:25.261926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:25.261928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:25.262195 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:25.262423 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:25.262677 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 10:35:25.262818 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:35:25.265811 14463 heartbeater.cc:503] Master 127.0.97.62:43783 was elected leader, sending a full tablet report...
I20260925 10:35:25.266994 14274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:35:25.273432 14274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:35:25.295027 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.284541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.0.97.1:33633)
0925 10:35:25.285328 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:25.285337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:25.285845 (+ 508us) server_negotiation.cc:407] Connection header received
0925 10:35:25.286230 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:25.286241 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:25.286527 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:25.286699 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:25.287714 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.289341 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.292240 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.293741 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.293862 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.294100 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:25.294526 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 10:35:25.294704 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":456,"threads_started":1}
I20260925 10:35:25.295143 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.284496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33633 (local address 127.0.0.1:55232)
0925 10:35:25.285691 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:25.285737 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:25.285991 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:25.287009 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:25.287016 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:25.287042 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:25.287490 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:25.287503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.289504 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.289509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:25.292036 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:25.292053 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.294059 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.294067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:25.294261 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.294334 (+ 73us) client_negotiation.cc:769] Sending connection context
0925 10:35:25.294475 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 10:35:25.294661 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":1,"thread_start_us":791,"threads_started":1}
I20260925 10:35:25.295913 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.285151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.0.97.2:37349)
0925 10:35:25.285718 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:25.285728 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:25.286687 (+ 959us) server_negotiation.cc:407] Connection header received
0925 10:35:25.286958 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:25.286965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:25.287228 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:25.287408 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:25.288408 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.289828 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.292919 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.294269 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.294419 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.294614 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:25.295487 (+ 873us) server_negotiation.cc:299] Negotiation successful
0925 10:35:25.295668 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":284,"threads_started":1}
I20260925 10:35:25.296586 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.284603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37349 (local address 127.0.0.1:55152)
0925 10:35:25.286583 (+ 1980us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:25.286603 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:25.286783 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:25.287694 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:25.287698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:25.287722 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:25.288195 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:25.288207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.290015 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.290020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:25.292183 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:25.292198 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.294759 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.294765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:25.294992 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.295080 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 10:35:25.295200 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 10:35:25.295328 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1852,"mutex_wait_us":30,"thread_start_us":1732,"threads_started":1}
I20260925 10:35:25.299202 14398 tablet_service.cc:1467] Processing CreateTablet for tablet 47f7575af913480fa3b71e21ac3c928b (DEFAULT_TABLE table=test-table [id=e3296cd6bc674c0aa13902d623682463]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:35:25.300050 14532 tablet_service.cc:1467] Processing CreateTablet for tablet 47f7575af913480fa3b71e21ac3c928b (DEFAULT_TABLE table=test-table [id=e3296cd6bc674c0aa13902d623682463]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:35:25.300693 14398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f7575af913480fa3b71e21ac3c928b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:25.301334 14532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f7575af913480fa3b71e21ac3c928b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:25.302171 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.285869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40949 (local address 127.0.0.1:53908)
0925 10:35:25.292652 (+ 6783us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:25.292672 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:25.292789 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:25.294868 (+ 2079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:25.294873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:25.294898 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:25.296167 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:25.296181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.298445 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.298450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:25.299787 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:25.299801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.301343 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.301349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:25.301517 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.301560 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 10:35:25.301673 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 10:35:25.301794 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6668,"mutex_wait_us":29,"thread_start_us":6536,"threads_started":1}
I20260925 10:35:25.302191 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.286202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.0.97.3:40949)
0925 10:35:25.294052 (+ 7850us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:25.294060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:25.294088 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:35:25.294176 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:25.294184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:25.294429 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:25.294609 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:25.296362 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.298281 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.299974 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.301131 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.301295 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.301519 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:25.301726 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 10:35:25.301870 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7678,"thread_start_us":7534,"threads_started":1}
I20260925 10:35:25.305079 14665 tablet_service.cc:1467] Processing CreateTablet for tablet 47f7575af913480fa3b71e21ac3c928b (DEFAULT_TABLE table=test-table [id=e3296cd6bc674c0aa13902d623682463]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:35:25.306620 14665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f7575af913480fa3b71e21ac3c928b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:25.315056 14748 tablet_bootstrap.cc:492] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed: Bootstrap starting.
I20260925 10:35:25.317283 14749 tablet_bootstrap.cc:492] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6: Bootstrap starting.
I20260925 10:35:25.319711 14748 tablet_bootstrap.cc:654] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:25.320772 14749 tablet_bootstrap.cc:654] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:25.320879 14748 log.cc:824] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:25.321058 14750 tablet_bootstrap.cc:492] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1: Bootstrap starting.
I20260925 10:35:25.321929 14749 log.cc:824] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:25.324608 14749 tablet_bootstrap.cc:492] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6: No bootstrap required, opened a new log
I20260925 10:35:25.324702 14750 tablet_bootstrap.cc:654] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:25.324879 14749 ts_tablet_manager.cc:1397] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260925 10:35:25.325480 14748 tablet_bootstrap.cc:492] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed: No bootstrap required, opened a new log
I20260925 10:35:25.325770 14748 ts_tablet_manager.cc:1397] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260925 10:35:25.325922 14750 log.cc:824] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:25.328696 14750 tablet_bootstrap.cc:492] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1: No bootstrap required, opened a new log
I20260925 10:35:25.328928 14750 ts_tablet_manager.cc:1397] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260925 10:35:25.332482 14749 raft_consensus.cc:348] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:25.332855 14749 raft_consensus.cc:374] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:25.332959 14749 raft_consensus.cc:729] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e146c068db545d596246e5e6cbf7fd6, State: Initialized, Role: FOLLOWER
I20260925 10:35:25.333564 14749 consensus_queue.cc:260] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:25.334813 14597 heartbeater.cc:503] Master 127.0.97.62:43783 was elected leader, sending a full tablet report...
I20260925 10:35:25.335420 14749 ts_tablet_manager.cc:1428] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
I20260925 10:35:25.335851 14748 raft_consensus.cc:348] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:25.336293 14748 raft_consensus.cc:374] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:25.336426 14748 raft_consensus.cc:729] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc3d44bdb2eb4acabbe96820ce4db1ed, State: Initialized, Role: FOLLOWER
I20260925 10:35:25.336999 14748 consensus_queue.cc:260] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:25.338596 14748 ts_tablet_manager.cc:1428] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed: Time spent starting tablet: real 0.013s user 0.008s sys 0.007s
I20260925 10:35:25.341204 14750 raft_consensus.cc:348] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:25.341570 14750 raft_consensus.cc:374] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:25.341674 14750 raft_consensus.cc:729] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f4d65d8f3ef48eeae930d5b0531ffa1, State: Initialized, Role: FOLLOWER
I20260925 10:35:25.342234 14750 consensus_queue.cc:260] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:25.346977 14730 heartbeater.cc:503] Master 127.0.97.62:43783 was elected leader, sending a full tablet report...
I20260925 10:35:25.347460 14750 ts_tablet_manager.cc:1428] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
I20260925 10:35:25.393734 14755 raft_consensus.cc:482] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:35:25.394038 14755 raft_consensus.cc:504] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:25.395752 14755 leader_election.cc:288] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e146c068db545d596246e5e6cbf7fd6 (127.0.97.2:37349), 6f4d65d8f3ef48eeae930d5b0531ffa1 (127.0.97.3:40949)
I20260925 10:35:25.401968 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.396198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39201 (local address 127.0.97.3:40949)
0925 10:35:25.396378 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:25.396386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:25.396883 (+ 497us) server_negotiation.cc:407] Connection header received
0925 10:35:25.397114 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:25.397119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:25.397201 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:25.397295 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:25.397815 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.398605 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.400364 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.401102 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.401239 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.401348 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:25.401605 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 10:35:25.401723 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:35:25.402031 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.395993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40949 (local address 127.0.97.1:39201)
0925 10:35:25.396771 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:25.396786 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:25.396974 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:25.397309 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:25.397314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:25.397336 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:25.397672 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:25.397685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.398743 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.398751 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:25.400215 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:25.400226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.401301 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.401306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:25.401448 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.401474 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 10:35:25.401581 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 10:35:25.401702 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":475,"threads_started":1}
I20260925 10:35:25.402184 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.396132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59235 (local address 127.0.97.2:37349)
0925 10:35:25.396358 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:25.396366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:25.397002 (+ 636us) server_negotiation.cc:407] Connection header received
0925 10:35:25.397193 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:25.397197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:25.397254 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:25.397348 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:25.397984 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.398751 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.400444 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.401298 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.401440 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.401545 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:25.401806 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 10:35:25.401920 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:35:25.402503 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.396106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37349 (local address 127.0.97.1:59235)
0925 10:35:25.396903 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:25.396916 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:25.397018 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:25.397363 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:25.397367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:25.397391 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:25.397820 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:25.397830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.398935 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.398939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:25.400302 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:25.400313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.401518 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.401524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:25.401663 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.401684 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 10:35:25.401782 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 10:35:25.401897 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":502,"threads_started":1}
I20260925 10:35:25.403196 14685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f7575af913480fa3b71e21ac3c928b" candidate_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" is_pre_election: true
I20260925 10:35:25.403496 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f7575af913480fa3b71e21ac3c928b" candidate_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e146c068db545d596246e5e6cbf7fd6" is_pre_election: true
I20260925 10:35:25.403787 14685 raft_consensus.cc:2454] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc3d44bdb2eb4acabbe96820ce4db1ed in term 0.
I20260925 10:35:25.404006 14552 raft_consensus.cc:2454] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc3d44bdb2eb4acabbe96820ce4db1ed in term 0.
I20260925 10:35:25.404547 14353 leader_election.cc:302] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f4d65d8f3ef48eeae930d5b0531ffa1, dc3d44bdb2eb4acabbe96820ce4db1ed; no voters:
I20260925 10:35:25.404964 14755 raft_consensus.cc:2781] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:35:25.405133 14755 raft_consensus.cc:482] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:35:25.405249 14755 raft_consensus.cc:3037] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:25.407532 14755 raft_consensus.cc:504] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:25.408166 14755 leader_election.cc:288] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [CANDIDATE]: Term 1 election: Requested vote from peers 7e146c068db545d596246e5e6cbf7fd6 (127.0.97.2:37349), 6f4d65d8f3ef48eeae930d5b0531ffa1 (127.0.97.3:40949)
I20260925 10:35:25.408442 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f7575af913480fa3b71e21ac3c928b" candidate_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e146c068db545d596246e5e6cbf7fd6"
I20260925 10:35:25.408583 14685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f7575af913480fa3b71e21ac3c928b" candidate_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1"
I20260925 10:35:25.408674 14552 raft_consensus.cc:3037] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:25.408803 14685 raft_consensus.cc:3037] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:25.410988 14552 raft_consensus.cc:2454] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc3d44bdb2eb4acabbe96820ce4db1ed in term 1.
I20260925 10:35:25.411023 14685 raft_consensus.cc:2454] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc3d44bdb2eb4acabbe96820ce4db1ed in term 1.
I20260925 10:35:25.411348 14354 leader_election.cc:302] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e146c068db545d596246e5e6cbf7fd6, dc3d44bdb2eb4acabbe96820ce4db1ed; no voters:
I20260925 10:35:25.411630 14755 raft_consensus.cc:2781] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:35:25.412004 14755 raft_consensus.cc:686] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [term 1 LEADER]: Becoming Leader. State: Replica: dc3d44bdb2eb4acabbe96820ce4db1ed, State: Running, Role: LEADER
I20260925 10:35:25.412432 14755 consensus_queue.cc:237] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:25.416900 14274 catalog_manager.cc:5495] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed reported cstate change: term changed from 0 to 1, leader changed from <none> to dc3d44bdb2eb4acabbe96820ce4db1ed (127.0.97.1). New cstate: current_term: 1 leader_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } health_report { overall_health: UNKNOWN } } }
I20260925 10:35:25.456352 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.451452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.0.97.1:33633)
0925 10:35:25.451649 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:25.451653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:25.451669 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:35:25.451753 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:25.451757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:25.451831 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:25.451963 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:35:25.452265 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.452967 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.454349 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.454761 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.454852 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.455293 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:35:25.455893 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:35:25.456001 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:25.456107 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 10:35:25.456169 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 10:35:25.464082 14731 tablet.cc:2369] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:35:25.464478 14552 raft_consensus.cc:1261] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6 [term 1 FOLLOWER]: Refusing update from remote peer dc3d44bdb2eb4acabbe96820ce4db1ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:35:25.464552 14685 raft_consensus.cc:1261] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 1 FOLLOWER]: Refusing update from remote peer dc3d44bdb2eb4acabbe96820ce4db1ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:35:25.465459 14755 consensus_queue.cc:1035] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:35:25.465742 14759 consensus_queue.cc:1035] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:35:25.469542 14598 tablet.cc:2369] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:35:25.481014 14762 mvcc.cc:205] Tried to move back new op lower bound from 7333202024293117952 to 7333202024093417472. Current Snapshot: MvccSnapshot[applied={T|T < 7333202024293117952}]
I20260925 10:35:25.482721 14763 mvcc.cc:205] Tried to move back new op lower bound from 7333202024293117952 to 7333202024093417472. Current Snapshot: MvccSnapshot[applied={T|T < 7333202024293117952}]
I20260925 10:35:25.484858 14764 mvcc.cc:205] Tried to move back new op lower bound from 7333202024293117952 to 7333202024093417472. Current Snapshot: MvccSnapshot[applied={T|T < 7333202024293117952}]
I20260925 10:35:25.495906 14274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:35:25.496122 14274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:35:25.499202 14274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f: Sending DeleteTablet for 3 replicas of tablet 47f7575af913480fa3b71e21ac3c928b
I20260925 10:35:25.500164 14398 tablet_service.cc:1514] Processing DeleteTablet for tablet 47f7575af913480fa3b71e21ac3c928b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:35:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55232
I20260925 10:35:25.500322 14665 tablet_service.cc:1514] Processing DeleteTablet for tablet 47f7575af913480fa3b71e21ac3c928b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:35:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53908
I20260925 10:35:25.500313 14532 tablet_service.cc:1514] Processing DeleteTablet for tablet 47f7575af913480fa3b71e21ac3c928b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:35:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55152
I20260925 10:35:25.502179 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 14335
I20260925 10:35:25.512579 14774 tablet_replica.cc:330] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6: stopping tablet replica
I20260925 10:35:25.513082 14774 raft_consensus.cc:2229] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:35:25.513669 14774 raft_consensus.cc:2258] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:35:25.516155 14774 ts_tablet_manager.cc:1905] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:35:25.517210 14773 tablet_replica.cc:330] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1: stopping tablet replica
I20260925 10:35:25.517604 14773 raft_consensus.cc:2229] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:35:25.518066 14773 raft_consensus.cc:2258] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:35:25.523319 14774 ts_tablet_manager.cc:1918] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:35:25.523440 14773 ts_tablet_manager.cc:1905] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:35:25.523478 14774 log.cc:1196] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/47f7575af913480fa3b71e21ac3c928b
I20260925 10:35:25.524000 14774 ts_tablet_manager.cc:1939] T 47f7575af913480fa3b71e21ac3c928b P 7e146c068db545d596246e5e6cbf7fd6: Deleting consensus metadata
I20260925 10:35:25.525728 14261 catalog_manager.cc:4841] TS 7e146c068db545d596246e5e6cbf7fd6 (127.0.97.2:37349): tablet 47f7575af913480fa3b71e21ac3c928b (table test-table [id=e3296cd6bc674c0aa13902d623682463]) successfully deleted
I20260925 10:35:25.526726 14772 tablet_replica.cc:330] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed: stopping tablet replica
I20260925 10:35:25.527169 14772 raft_consensus.cc:2229] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:35:25.527818 14772 raft_consensus.cc:2258] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:35:25.529834 14772 ts_tablet_manager.cc:1905] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:35:25.531447 14773 ts_tablet_manager.cc:1918] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:35:25.531600 14773 log.cc:1196] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/47f7575af913480fa3b71e21ac3c928b
Sep 25 10:35:25 dist-test-slave-dxdd krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:35:25.532119 14773 ts_tablet_manager.cc:1939] T 47f7575af913480fa3b71e21ac3c928b P 6f4d65d8f3ef48eeae930d5b0531ffa1: Deleting consensus metadata
I20260925 10:35:25.533910 14260 catalog_manager.cc:4841] TS 6f4d65d8f3ef48eeae930d5b0531ffa1 (127.0.97.3:40949): tablet 47f7575af913480fa3b71e21ac3c928b (table test-table [id=e3296cd6bc674c0aa13902d623682463]) successfully deleted
I20260925 10:35:25.538113 14772 ts_tablet_manager.cc:1918] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:35:25.538292 14772 log.cc:1196] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/47f7575af913480fa3b71e21ac3c928b
I20260925 10:35:25.538920 14772 ts_tablet_manager.cc:1939] T 47f7575af913480fa3b71e21ac3c928b P dc3d44bdb2eb4acabbe96820ce4db1ed: Deleting consensus metadata
I20260925 10:35:25.540714 14258 catalog_manager.cc:4841] TS dc3d44bdb2eb4acabbe96820ce4db1ed (127.0.97.1:33633): tablet 47f7575af913480fa3b71e21ac3c928b (table test-table [id=e3296cd6bc674c0aa13902d623682463]) successfully deleted
I20260925 10:35:25.541055 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.503567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55250 (local address 127.0.97.1:33633)
0925 10:35:25.503760 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:25.503769 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:25.514520 (+ 10751us) server_negotiation.cc:407] Connection header received
0925 10:35:25.514804 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:25.514809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:25.514884 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:25.514975 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:25.518472 (+ 3497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.519709 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.526256 (+ 6547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.527044 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.527183 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.533161 (+ 5978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:25.533175 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:25.533192 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:25.533241 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:25.537540 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:25.538500 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:25.538516 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:25.538523 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:25.538737 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:25.539192 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:25.539199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:25.539202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:25.539607 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:25.539942 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:25.540275 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 10:35:25.540521 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:35:25.541755 14458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:25.640408 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 14335
I20260925 10:35:25.659260 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 14469
Sep 25 10:35:25 dist-test-slave-dxdd krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:35:25.675515 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.660047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.0.97.2:37349)
0925 10:35:25.660250 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:25.660256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:25.660275 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:35:25.660479 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:25.660483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:25.660546 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:25.660674 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:25.661907 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.662701 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.664149 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.664553 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.664686 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.669040 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:25.669057 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:25.669078 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:25.669120 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:25.672822 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:25.673531 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:25.673541 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:25.673545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:25.673681 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:25.674004 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:25.674009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:25.674013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:25.674493 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:25.674773 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:25.674957 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 10:35:25.675172 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:35:25.676219 14592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:25.765115 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 14469
I20260925 10:35:25.781711 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 14602
Sep 25 10:35:25 dist-test-slave-dxdd krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:35:25.796669 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:25.782522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53922 (local address 127.0.97.3:40949)
0925 10:35:25.782687 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:25.782691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:25.782737 (+ 46us) server_negotiation.cc:407] Connection header received
0925 10:35:25.782932 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:25.782937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:25.783001 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:25.783081 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:25.784409 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.785130 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.786582 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:25.787011 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:25.787109 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:25.791147 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:25.791163 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:25.791179 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:25.791215 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:25.793959 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:25.794602 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:25.794610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:25.794612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:25.794754 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:25.795071 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:25.795074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:25.795076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:25.795415 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:25.795750 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:25.796128 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 10:35:25.796365 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:35:25.797307 14725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:25.885407 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 14602
I20260925 10:35:25.902463 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 14242
I20260925 10:35:25.903568 14304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:26.015508 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 14242
I20260925 10:35:26.036314 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43783
--webserver_interface=127.0.97.62
--webserver_port=36329
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:35:26.357200 14780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:26.357461 14780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:26.357544 14780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:26.364324 14780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:35:26.364416 14780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:26.364459 14780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:26.364516 14780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:35:26.364553 14780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:35:26.373934 14780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:26.375098 14780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:26.376297 14780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:26.397657 14786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:26.397696 14789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:26.397756 14787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:26.399715 14780 server_base.cc:1034] running on GCE node
I20260925 10:35:26.400453 14780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:26.401795 14780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:26.403020 14780 hybrid_clock.cc:648] HybridClock initialized: now 1790332526402969 us; error 43 us; skew 500 ppm
I20260925 10:35:26.403362 14780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:26 dist-test-slave-dxdd krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:26.406770 14780 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:35:26.408430 14780 webserver.cc:460] Webserver started at http://127.0.97.62:36329/ using document root <none> and password file <none>
I20260925 10:35:26.409171 14780 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:26.409322 14780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:26.415025 14780 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:35:26.417857 14796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:26.419195 14780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:35:26.419377 14780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "6b0d39c466504eef8ec3e7d55023612f"
format_stamp: "Formatted at 2026-09-25 10:35:23 on dist-test-slave-dxdd"
server_key: "b8c18ba86bea496b51b7e392d73a436d"
server_key_iv: "99e6e78019097f25ff7a19bf803323ee"
server_key_version: "encryptionkey@0"
I20260925 10:35:26.420415 14780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:26.444658 14780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:26.448462 14780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:26.448746 14780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:26.467427 14780 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43783
I20260925 10:35:26.467453 14847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43783 every 8 connection(s)
I20260925 10:35:26.468897 14780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:35:26.475518 14848 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:35:26.478171 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 14780
I20260925 10:35:26.478672 14848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f: Bootstrap starting.
I20260925 10:35:26.479065 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33633
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43783
--webserver_port=34191
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:35:26.503103 14848 log.cc:824] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:26.517953 14848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:35:26.518606 14848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f: Bootstrap complete.
I20260925 10:35:26.531572 14848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } }
I20260925 10:35:26.532691 14848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0d39c466504eef8ec3e7d55023612f, State: Initialized, Role: FOLLOWER
I20260925 10:35:26.533372 14848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } }
I20260925 10:35:26.533721 14848 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:35:26.533855 14848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:35:26.534039 14848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:35:26.539773 14848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } }
I20260925 10:35:26.540351 14848 leader_election.cc:302] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b0d39c466504eef8ec3e7d55023612f; no voters:
I20260925 10:35:26.540974 14848 leader_election.cc:288] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:35:26.541110 14852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:35:26.542423 14852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 2 LEADER]: Becoming Leader. State: Replica: 6b0d39c466504eef8ec3e7d55023612f, State: Running, Role: LEADER
I20260925 10:35:26.542932 14852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } }
I20260925 10:35:26.543145 14848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:35:26.545245 14854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b0d39c466504eef8ec3e7d55023612f. Latest consensus state: current_term: 2 leader_uuid: "6b0d39c466504eef8ec3e7d55023612f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } } }
I20260925 10:35:26.545642 14854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:26.546218 14853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6b0d39c466504eef8ec3e7d55023612f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } } }
I20260925 10:35:26.546581 14853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:26.548806 14858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:35:26.565876 14858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:35:26.567767 14858 catalog_manager.cc:1259] Loaded cluster ID: bb3956981dec43f3bb31901ee892836f
I20260925 10:35:26.567968 14858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:35:26.572031 14858 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 10:35:26.597944 14858 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 10:35:26.618002 14858 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:35:26.619794 14858 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f: Loaded TSK: 0
I20260925 10:35:26.621279 14858 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:35:26.847295 14850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:26.847585 14850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:26.847745 14850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:26.854935 14850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:26.855077 14850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:26.855247 14850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:35:26.864566 14850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34191
--tserver_master_addrs=127.0.97.62:43783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:26.865878 14850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:26.867213 14850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:26.881259 14876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:26.881565 14877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:26.881800 14879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:26.883016 14850 server_base.cc:1034] running on GCE node
I20260925 10:35:26.883582 14850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:26.884500 14850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:26.885728 14850 hybrid_clock.cc:648] HybridClock initialized: now 1790332526885671 us; error 53 us; skew 500 ppm
I20260925 10:35:26.886025 14850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:26 dist-test-slave-dxdd krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:26.889317 14850 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:35:26.890808 14850 webserver.cc:460] Webserver started at http://127.0.97.1:34191/ using document root <none> and password file <none>
I20260925 10:35:26.891508 14850 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:26.891660 14850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:26.897010 14850 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 10:35:26.899713 14886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:26.901001 14850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:35:26.901226 14850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed"
format_stamp: "Formatted at 2026-09-25 10:35:24 on dist-test-slave-dxdd"
server_key: "742019e08499c25a682dfa77ef966d88"
server_key_iv: "5df8b1cc2fd5197bdc21cd60a5ab5f20"
server_key_version: "encryptionkey@0"
I20260925 10:35:26.902217 14850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:26.916479 14850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:26.920637 14850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:26.920892 14850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:26.921779 14850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:26.923040 14850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:26.923122 14850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:26.923261 14850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:26.923368 14850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:26.956452 14850 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33633
I20260925 10:35:26.956527 14998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33633 every 8 connection(s)
I20260925 10:35:26.957835 14850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:35:26.965667 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 14850
I20260925 10:35:26.966444 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37349
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43783
--webserver_port=36013
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:35:26 dist-test-slave-dxdd krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:26.982669 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:26.960603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58483 (local address 127.0.97.62:43783)
0925 10:35:26.961203 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:26.961215 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:26.961823 (+ 608us) server_negotiation.cc:407] Connection header received
0925 10:35:26.963919 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:26.963933 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:26.964238 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:26.964527 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:26.966221 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:26.967554 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:26.970027 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:26.970711 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:26.970845 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:26.975526 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:26.975533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:26.975546 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:26.975581 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:26.978352 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:26.979239 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:26.979245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:26.979251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:26.979397 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:26.979889 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:26.979899 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:26.979901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:26.980309 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:26.980446 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:26.981325 (+ 879us) server_negotiation.cc:299] Negotiation successful
0925 10:35:26.981673 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":273,"threads_started":1}
I20260925 10:35:26.983048 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:26.960858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:58483)
0925 10:35:26.961647 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:26.961696 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:26.963435 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:26.964656 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:26.964662 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:26.965223 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:26.966002 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:26.966026 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:26.967709 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:26.967717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:26.969876 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:26.969892 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:26.970959 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:26.970968 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:26.971300 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:26.972024 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:26.972065 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:26.975269 (+ 3204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:26.978493 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:26.978504 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:26.978520 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:26.979123 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:26.979523 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:26.979531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:26.979537 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:26.979783 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:26.980461 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:26.980471 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:26.980885 (+ 414us) client_negotiation.cc:769] Sending connection context
0925 10:35:26.981173 (+ 288us) client_negotiation.cc:240] Negotiation successful
0925 10:35:26.981760 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":345,"threads_started":1}
I20260925 10:35:26.985986 14999 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43783
I20260925 10:35:26.986277 14999 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:26.986974 14999 heartbeater.cc:511] Master 127.0.97.62:43783 requested a full tablet report, sending...
I20260925 10:35:26.989347 14813 ts_manager.cc:194] Registered new tserver with Master: dc3d44bdb2eb4acabbe96820ce4db1ed (127.0.97.1:33633)
I20260925 10:35:26.991703 14813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58483
W20260925 10:35:27.291985 15003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:27.292249 15003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:27.292339 15003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:27.299383 15003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:27.299476 15003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:27.299573 15003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:35:27.308595 15003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36013
--tserver_master_addrs=127.0.97.62:43783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:27.309669 15003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:27.310966 15003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:27.323550 15008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:27.323530 15009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:27.324148 15011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:27.324786 15003 server_base.cc:1034] running on GCE node
I20260925 10:35:27.325470 15003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:27.326512 15003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:27.327745 15003 hybrid_clock.cc:648] HybridClock initialized: now 1790332527327713 us; error 46 us; skew 500 ppm
I20260925 10:35:27.328042 15003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:27 dist-test-slave-dxdd krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:27.331032 15003 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:35:27.332330 15003 webserver.cc:460] Webserver started at http://127.0.97.2:36013/ using document root <none> and password file <none>
I20260925 10:35:27.332958 15003 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:27.333123 15003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:27.338025 15003 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260925 10:35:27.340626 15018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:27.341794 15003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:35:27.341979 15003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "7e146c068db545d596246e5e6cbf7fd6"
format_stamp: "Formatted at 2026-09-25 10:35:24 on dist-test-slave-dxdd"
server_key: "4aadb17a7a3e107a604fdcf784b6e10e"
server_key_iv: "a9364526c51d3b8c1263d31b37d132ec"
server_key_version: "encryptionkey@0"
I20260925 10:35:27.342973 15003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:27.359915 15003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:27.363541 15003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:27.363806 15003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:27.364724 15003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:27.365936 15003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:27.366078 15003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:27.366201 15003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:27.366289 15003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:27.398579 15003 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37349
I20260925 10:35:27.398669 15130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37349 every 8 connection(s)
I20260925 10:35:27.399947 15003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:35:27.401412 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 15003
I20260925 10:35:27.402114 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40949
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43783
--webserver_port=39733
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:35:27 dist-test-slave-dxdd krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:27.423322 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:27.402641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46179 (local address 127.0.97.62:43783)
0925 10:35:27.402834 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:27.402839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:27.404164 (+ 1325us) server_negotiation.cc:407] Connection header received
0925 10:35:27.406963 (+ 2799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:27.406967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:27.407033 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:27.407118 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:27.409089 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.410022 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.412321 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.412726 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.412821 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:27.417266 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:27.417271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:27.417274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:27.417306 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:27.419969 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:27.420949 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:27.420953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:27.420955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:27.421076 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:27.421571 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:27.421592 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:27.421594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:27.421837 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:27.421954 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:27.422915 (+ 961us) server_negotiation.cc:299] Negotiation successful
0925 10:35:27.423088 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 10:35:27.424997 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:27.403169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:46179)
0925 10:35:27.403982 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:27.404028 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:27.406684 (+ 2656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:27.407385 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:27.407398 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:27.408155 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:27.408920 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:27.408946 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.410174 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.410178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:27.412182 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:27.412198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.412949 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.412958 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:27.413303 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:27.414026 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:27.414062 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:27.417055 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:27.420110 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:27.420121 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:27.420142 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:27.420819 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:27.421184 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:27.421191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:27.421197 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:27.421455 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:27.421965 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:27.421976 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:27.422479 (+ 503us) client_negotiation.cc:769] Sending connection context
0925 10:35:27.422840 (+ 361us) client_negotiation.cc:240] Negotiation successful
0925 10:35:27.423437 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":308,"threads_started":1}
I20260925 10:35:27.426528 15131 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43783
I20260925 10:35:27.426817 15131 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:27.427485 15131 heartbeater.cc:511] Master 127.0.97.62:43783 requested a full tablet report, sending...
I20260925 10:35:27.428834 14813 ts_manager.cc:194] Registered new tserver with Master: 7e146c068db545d596246e5e6cbf7fd6 (127.0.97.2:37349)
I20260925 10:35:27.429605 14813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46179
W20260925 10:35:27.721014 15134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:27.721314 15134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:27.721493 15134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:27.728499 15134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:27.728639 15134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:27.728806 15134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:35:27.738166 15134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39733
--tserver_master_addrs=127.0.97.62:43783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:27.739396 15134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:27.740654 15134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:27.753302 15139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:27.753401 15140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:27.753369 15142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:27.754248 15134 server_base.cc:1034] running on GCE node
I20260925 10:35:27.754802 15134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:27.755699 15134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:27.756924 15134 hybrid_clock.cc:648] HybridClock initialized: now 1790332527756866 us; error 51 us; skew 500 ppm
I20260925 10:35:27.757220 15134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:27 dist-test-slave-dxdd krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:27.760219 15134 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:35:27.761525 15134 webserver.cc:460] Webserver started at http://127.0.97.3:39733/ using document root <none> and password file <none>
I20260925 10:35:27.762149 15134 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:27.762349 15134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:27.767303 15134 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 10:35:27.769754 15149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:27.771023 15134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:35:27.771200 15134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1"
format_stamp: "Formatted at 2026-09-25 10:35:25 on dist-test-slave-dxdd"
server_key: "95e9a4c48d4dec08c6b290347a2e710e"
server_key_iv: "7520057a35ce30d2dd607b0ef929c575"
server_key_version: "encryptionkey@0"
I20260925 10:35:27.772125 15134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:27.790143 15134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:27.794667 15134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:27.794934 15134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:27.795861 15134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:27.797184 15134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:27.797314 15134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:27.797454 15134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:27.797544 15134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:27.830760 15134 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40949
I20260925 10:35:27.830811 15261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40949 every 8 connection(s)
I20260925 10:35:27.832155 15134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:35:27.837280 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 15134
Sep 25 10:35:27 dist-test-slave-dxdd krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:27.855473 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:27.838241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55822 (local address 127.0.97.62:43783)
0925 10:35:27.839053 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:27.839059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:27.839076 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:35:27.839130 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:27.839135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:27.839180 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:27.839300 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:27.841032 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.842125 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.843722 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.844490 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.844579 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:27.847671 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:27.847680 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:27.847685 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:27.847721 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:27.852463 (+ 4742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:27.853254 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:27.853260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:27.853263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:27.853449 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:27.853886 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:27.853889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:27.853891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:27.854352 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:27.854591 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:27.854904 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 10:35:27.855054 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":699,"spinlock_wait_cycles":11776,"thread_start_us":579,"threads_started":1}
I20260925 10:35:27.855845 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:27.834591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53225 (local address 127.0.97.62:43783)
0925 10:35:27.834828 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:27.834835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:27.835936 (+ 1101us) server_negotiation.cc:407] Connection header received
0925 10:35:27.838059 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:27.838066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:27.838157 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:27.838261 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:27.839925 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.841080 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.842970 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.843616 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.843778 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:27.847692 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:27.847696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:27.847700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:27.847739 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:27.852397 (+ 4658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:27.853254 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:27.853260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:27.853263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:27.853450 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:27.853961 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:27.853965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:27.853967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:27.854338 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:27.854498 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:27.855297 (+ 799us) server_negotiation.cc:299] Negotiation successful
0925 10:35:27.855507 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":85,"spinlock_wait_cycles":13952}
I20260925 10:35:27.856976 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:27.835135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:53225)
0925 10:35:27.835765 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:27.835799 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:27.837750 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:27.838600 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:27.838616 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:27.839240 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:27.839746 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:27.839762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.841236 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.841241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:27.842822 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:27.842833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.843838 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.843848 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:27.844129 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:27.844883 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:27.844916 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:27.847509 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:27.852571 (+ 5062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:27.852579 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:27.852593 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:27.853143 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:27.853568 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:27.853573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:27.853577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:27.853848 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:27.854467 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:27.854474 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:27.854876 (+ 402us) client_negotiation.cc:769] Sending connection context
0925 10:35:27.855166 (+ 290us) client_negotiation.cc:240] Negotiation successful
0925 10:35:27.855691 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":243,"threads_started":1}
I20260925 10:35:27.858502 15262 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43783
I20260925 10:35:27.858806 15262 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:27.859476 15262 heartbeater.cc:511] Master 127.0.97.62:43783 requested a full tablet report, sending...
I20260925 10:35:27.860819 14812 ts_manager.cc:194] Registered new tserver with Master: 6f4d65d8f3ef48eeae930d5b0531ffa1 (127.0.97.3:40949)
I20260925 10:35:27.861668 14812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53225
I20260925 10:35:27.870164 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:35:27.876814 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:27.871045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.0.97.62:43783)
0925 10:35:27.871196 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:27.871200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:27.871620 (+ 420us) server_negotiation.cc:407] Connection header received
0925 10:35:27.871842 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:27.871845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:27.871969 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:27.872122 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:35:27.872550 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.873214 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.874618 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.875049 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.875192 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:27.875707 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:35:27.876172 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:35:27.876329 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:27.876428 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 10:35:27.876502 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:35:27.877600 14812 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:35:27.884567 14812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:35:27.915741 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:27.895885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.0.97.2:37349)
0925 10:35:27.908663 (+ 12778us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:27.908673 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:27.908702 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:35:27.908790 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:27.908799 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:27.909014 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:27.909618 (+ 604us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:27.909971 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.911135 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.913285 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.914477 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.914613 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:27.914842 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:27.915246 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 10:35:27.915398 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12615,"thread_start_us":12489,"threads_started":1}
I20260925 10:35:27.915792 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:27.895788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37349 (local address 127.0.0.1:55174)
0925 10:35:27.906997 (+ 11209us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:27.907013 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:27.907135 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:27.909237 (+ 2102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:27.909241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:27.909270 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:27.909813 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:27.909826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.911317 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.911327 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:27.913133 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:27.913144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.914880 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.914898 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:27.915025 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:27.915069 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 10:35:27.915198 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 10:35:27.915370 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11089,"mutex_wait_us":103,"thread_start_us":12263,"threads_started":1}
I20260925 10:35:27.916848 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:27.895727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33633 (local address 127.0.0.1:55266)
0925 10:35:27.906332 (+ 10605us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:27.906369 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:27.906547 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:27.908725 (+ 2178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:27.908733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:27.908764 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:27.909231 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:27.909243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.911182 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.911190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:27.913811 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:27.913825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.916115 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.916121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:27.916299 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:27.916344 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 10:35:27.916421 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 10:35:27.916512 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10361,"mutex_wait_us":108,"thread_start_us":11003,"threads_started":1}
I20260925 10:35:27.917248 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:27.896509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53928 (local address 127.0.97.3:40949)
0925 10:35:27.907881 (+ 11372us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:27.907891 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:27.908506 (+ 615us) server_negotiation.cc:407] Connection header received
0925 10:35:27.908610 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:27.908620 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:27.908834 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:27.909025 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:27.910185 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.911717 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.914181 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.916029 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.916146 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:27.916348 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:27.916744 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 10:35:27.916922 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11173,"thread_start_us":11026,"threads_started":1}
I20260925 10:35:27.917688 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:27.896262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.0.97.1:33633)
0925 10:35:27.908103 (+ 11841us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:27.908112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:27.908141 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:35:27.908241 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:27.908249 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:27.908505 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:27.908694 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:27.909427 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.911042 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.914249 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.915715 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.916922 (+ 1207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:27.917124 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:27.917190 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 10:35:27.917358 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11675,"thread_start_us":11577,"threads_started":1}
I20260925 10:35:27.918229 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:27.896149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40949 (local address 127.0.0.1:53928)
0925 10:35:27.908282 (+ 12133us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:27.908297 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:27.908396 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:27.909020 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:27.909024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:27.909050 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:27.909554 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:27.909566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.911881 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.911888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:27.914045 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:27.914056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.916394 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.916400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:27.916555 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:27.916584 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 10:35:27.916676 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 10:35:27.916805 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12037,"mutex_wait_us":61,"thread_start_us":9806,"threads_started":1}
I20260925 10:35:27.920118 15066 tablet_service.cc:1467] Processing CreateTablet for tablet f724920ba8fe454e8c68943fa5e23099 (DEFAULT_TABLE table=test-table [id=75b0d21e33d94c3f8bf9042f23b62339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:35:27.921103 15197 tablet_service.cc:1467] Processing CreateTablet for tablet f724920ba8fe454e8c68943fa5e23099 (DEFAULT_TABLE table=test-table [id=75b0d21e33d94c3f8bf9042f23b62339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:35:27.921573 15066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f724920ba8fe454e8c68943fa5e23099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:27.921489 14934 tablet_service.cc:1467] Processing CreateTablet for tablet f724920ba8fe454e8c68943fa5e23099 (DEFAULT_TABLE table=test-table [id=75b0d21e33d94c3f8bf9042f23b62339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:35:27.922185 15197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f724920ba8fe454e8c68943fa5e23099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:27.922662 14934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f724920ba8fe454e8c68943fa5e23099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:27.933637 15278 tablet_bootstrap.cc:492] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1: Bootstrap starting.
I20260925 10:35:27.936782 15279 tablet_bootstrap.cc:492] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed: Bootstrap starting.
I20260925 10:35:27.937489 15278 tablet_bootstrap.cc:654] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:27.937754 15280 tablet_bootstrap.cc:492] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6: Bootstrap starting.
I20260925 10:35:27.938973 15278 log.cc:824] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:27.940589 15279 tablet_bootstrap.cc:654] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:27.940619 15280 tablet_bootstrap.cc:654] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:27.941751 15280 log.cc:824] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:27.942132 15279 log.cc:824] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:27.942431 15278 tablet_bootstrap.cc:492] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1: No bootstrap required, opened a new log
I20260925 10:35:27.942713 15278 ts_tablet_manager.cc:1397] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 10:35:27.944620 15279 tablet_bootstrap.cc:492] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed: No bootstrap required, opened a new log
I20260925 10:35:27.944729 15280 tablet_bootstrap.cc:492] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6: No bootstrap required, opened a new log
I20260925 10:35:27.944917 15279 ts_tablet_manager.cc:1397] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 10:35:27.944983 15280 ts_tablet_manager.cc:1397] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260925 10:35:27.952301 15279 raft_consensus.cc:348] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:27.952651 15279 raft_consensus.cc:374] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:27.952785 15279 raft_consensus.cc:729] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc3d44bdb2eb4acabbe96820ce4db1ed, State: Initialized, Role: FOLLOWER
I20260925 10:35:27.953299 15279 consensus_queue.cc:260] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:27.953519 15278 raft_consensus.cc:348] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:27.953917 15278 raft_consensus.cc:374] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:27.954026 15278 raft_consensus.cc:729] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f4d65d8f3ef48eeae930d5b0531ffa1, State: Initialized, Role: FOLLOWER
I20260925 10:35:27.954509 14999 heartbeater.cc:503] Master 127.0.97.62:43783 was elected leader, sending a full tablet report...
I20260925 10:35:27.954689 15278 consensus_queue.cc:260] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:27.955039 15279 ts_tablet_manager.cc:1428] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed: Time spent starting tablet: real 0.010s user 0.001s sys 0.009s
I20260925 10:35:27.955500 15280 raft_consensus.cc:348] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:27.955912 15280 raft_consensus.cc:374] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:27.956007 15280 raft_consensus.cc:729] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e146c068db545d596246e5e6cbf7fd6, State: Initialized, Role: FOLLOWER
I20260925 10:35:27.956538 15280 consensus_queue.cc:260] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:27.957583 15131 heartbeater.cc:503] Master 127.0.97.62:43783 was elected leader, sending a full tablet report...
I20260925 10:35:27.958144 15280 ts_tablet_manager.cc:1428] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260925 10:35:27.959882 15000 tablet.cc:2369] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:35:27.967001 15262 heartbeater.cc:503] Master 127.0.97.62:43783 was elected leader, sending a full tablet report...
I20260925 10:35:27.967506 15278 ts_tablet_manager.cc:1428] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1: Time spent starting tablet: real 0.025s user 0.004s sys 0.012s
I20260925 10:35:27.986977 15285 raft_consensus.cc:482] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:35:27.987252 15285 raft_consensus.cc:504] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:27.988703 15285 leader_election.cc:288] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e146c068db545d596246e5e6cbf7fd6 (127.0.97.2:37349), dc3d44bdb2eb4acabbe96820ce4db1ed (127.0.97.1:33633)
I20260925 10:35:27.994182 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:27.989058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37349 (local address 127.0.97.3:38619)
0925 10:35:27.989353 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:27.989367 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:27.989522 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:27.989860 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:27.989864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:27.989889 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:27.990065 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:27.990074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.991078 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.991083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:27.992343 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:27.992355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.993474 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.993479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:27.993664 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:27.993687 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:35:27.993791 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:35:27.993913 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260925 10:35:27.994174 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:27.988978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38619 (local address 127.0.97.2:37349)
0925 10:35:27.989173 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:27.989180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:27.989448 (+ 268us) server_negotiation.cc:407] Connection header received
0925 10:35:27.989662 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:27.989669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:27.989732 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:27.989836 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:27.990200 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.990938 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.992496 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.993335 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.993467 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:27.993563 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:27.993818 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 10:35:27.993936 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:35:27.995028 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:27.989071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52817 (local address 127.0.97.1:33633)
0925 10:35:27.989271 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:27.989279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:27.989931 (+ 652us) server_negotiation.cc:407] Connection header received
0925 10:35:27.990173 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:27.990177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:27.990231 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:27.990345 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:27.990845 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.991528 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.993167 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.993981 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.994133 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:27.994223 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:27.994649 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 10:35:27.994749 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:35:27.995080 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:27.989105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33633 (local address 127.0.97.3:52817)
0925 10:35:27.989826 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:27.989839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:27.990028 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:27.990371 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:27.990375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:27.990407 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:27.990696 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:27.990703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.991673 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.991678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:27.993012 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:27.993023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:27.994192 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:27.994198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:27.994462 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:27.994513 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 10:35:27.994612 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:35:27.994749 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"spinlock_wait_cycles":19840,"thread_start_us":429,"threads_started":1}
I20260925 10:35:27.995321 15086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f724920ba8fe454e8c68943fa5e23099" candidate_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e146c068db545d596246e5e6cbf7fd6" is_pre_election: true
I20260925 10:35:27.995843 15086 raft_consensus.cc:2454] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f4d65d8f3ef48eeae930d5b0531ffa1 in term 0.
I20260925 10:35:27.995947 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f724920ba8fe454e8c68943fa5e23099" candidate_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" is_pre_election: true
I20260925 10:35:27.996475 14954 raft_consensus.cc:2454] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f4d65d8f3ef48eeae930d5b0531ffa1 in term 0.
I20260925 10:35:27.996519 15153 leader_election.cc:302] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f4d65d8f3ef48eeae930d5b0531ffa1, 7e146c068db545d596246e5e6cbf7fd6; no voters:
I20260925 10:35:27.996922 15285 raft_consensus.cc:2781] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:35:27.997088 15285 raft_consensus.cc:482] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:35:27.997200 15285 raft_consensus.cc:3037] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:27.999401 15285 raft_consensus.cc:504] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:27.999982 15285 leader_election.cc:288] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [CANDIDATE]: Term 1 election: Requested vote from peers 7e146c068db545d596246e5e6cbf7fd6 (127.0.97.2:37349), dc3d44bdb2eb4acabbe96820ce4db1ed (127.0.97.1:33633)
I20260925 10:35:28.000298 15086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f724920ba8fe454e8c68943fa5e23099" candidate_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e146c068db545d596246e5e6cbf7fd6"
I20260925 10:35:28.000413 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f724920ba8fe454e8c68943fa5e23099" candidate_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed"
I20260925 10:35:28.000479 15086 raft_consensus.cc:3037] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:28.000655 14954 raft_consensus.cc:3037] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:28.003043 15086 raft_consensus.cc:2454] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f4d65d8f3ef48eeae930d5b0531ffa1 in term 1.
I20260925 10:35:28.003163 14954 raft_consensus.cc:2454] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f4d65d8f3ef48eeae930d5b0531ffa1 in term 1.
I20260925 10:35:28.003401 15153 leader_election.cc:302] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f4d65d8f3ef48eeae930d5b0531ffa1, 7e146c068db545d596246e5e6cbf7fd6; no voters:
I20260925 10:35:28.003681 15285 raft_consensus.cc:2781] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:35:28.004099 15285 raft_consensus.cc:686] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 1 LEADER]: Becoming Leader. State: Replica: 6f4d65d8f3ef48eeae930d5b0531ffa1, State: Running, Role: LEADER
I20260925 10:35:28.004573 15285 consensus_queue.cc:237] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } } peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } }
I20260925 10:35:28.008841 14813 catalog_manager.cc:5495] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f4d65d8f3ef48eeae930d5b0531ffa1 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40949 } health_report { overall_health: HEALTHY } } }
I20260925 10:35:28.032651 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:28.027508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53930 (local address 127.0.97.3:40949)
0925 10:35:28.027681 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:28.027688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:28.027720 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:35:28.027808 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:28.027815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:28.027888 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:28.028031 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:35:28.028387 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:28.029117 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:28.030573 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:28.030968 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:28.031096 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:28.031618 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:35:28.032085 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:35:28.032243 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:28.032334 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 10:35:28.032407 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:35:28.040181 14954 raft_consensus.cc:1261] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed [term 1 FOLLOWER]: Refusing update from remote peer 6f4d65d8f3ef48eeae930d5b0531ffa1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:35:28.040220 15086 raft_consensus.cc:1261] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6 [term 1 FOLLOWER]: Refusing update from remote peer 6f4d65d8f3ef48eeae930d5b0531ffa1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:35:28.041116 15285 consensus_queue.cc:1035] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:35:28.041404 15288 consensus_queue.cc:1035] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e146c068db545d596246e5e6cbf7fd6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:35:28.057600 15291 mvcc.cc:205] Tried to move back new op lower bound from 7333202034843942912 to 7333202034711724032. Current Snapshot: MvccSnapshot[applied={T|T < 7333202034843942912}]
I20260925 10:35:28.059753 15292 mvcc.cc:205] Tried to move back new op lower bound from 7333202034843942912 to 7333202034711724032. Current Snapshot: MvccSnapshot[applied={T|T < 7333202034843942912}]
I20260925 10:35:28.075228 14811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:35:28.075484 14811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:35:28.079198 14811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f: Sending DeleteTablet for 3 replicas of tablet f724920ba8fe454e8c68943fa5e23099
I20260925 10:35:28.080402 14934 tablet_service.cc:1514] Processing DeleteTablet for tablet f724920ba8fe454e8c68943fa5e23099 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:35:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55266
I20260925 10:35:28.080458 15066 tablet_service.cc:1514] Processing DeleteTablet for tablet f724920ba8fe454e8c68943fa5e23099 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:35:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55174
I20260925 10:35:28.081107 15197 tablet_service.cc:1514] Processing DeleteTablet for tablet f724920ba8fe454e8c68943fa5e23099 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:35:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53928
I20260925 10:35:28.081327 15302 tablet_replica.cc:330] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed: stopping tablet replica
I20260925 10:35:28.081525 15303 tablet_replica.cc:330] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6: stopping tablet replica
I20260925 10:35:28.081844 15302 raft_consensus.cc:2229] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:35:28.081856 15303 raft_consensus.cc:2229] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:35:28.082336 15303 raft_consensus.cc:2258] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:35:28.082348 15302 raft_consensus.cc:2258] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:35:28.083171 15304 tablet_replica.cc:330] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1: stopping tablet replica
I20260925 10:35:28.083561 15304 raft_consensus.cc:2229] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:35:28.083716 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 14850
I20260925 10:35:28.085026 15304 raft_consensus.cc:2258] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:35:28.085371 15302 ts_tablet_manager.cc:1905] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:35:28.085758 15303 ts_tablet_manager.cc:1905] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:35:28.092893 15303 ts_tablet_manager.cc:1918] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:35:28.093046 15303 log.cc:1196] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/f724920ba8fe454e8c68943fa5e23099
I20260925 10:35:28.093580 15303 ts_tablet_manager.cc:1939] T f724920ba8fe454e8c68943fa5e23099 P 7e146c068db545d596246e5e6cbf7fd6: Deleting consensus metadata
I20260925 10:35:28.094205 15304 ts_tablet_manager.cc:1905] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:35:28.095331 14800 catalog_manager.cc:4841] TS 7e146c068db545d596246e5e6cbf7fd6 (127.0.97.2:37349): tablet f724920ba8fe454e8c68943fa5e23099 (table test-table [id=75b0d21e33d94c3f8bf9042f23b62339]) successfully deleted
I20260925 10:35:28.097411 15302 ts_tablet_manager.cc:1918] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:35:28.097662 15302 log.cc:1196] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/f724920ba8fe454e8c68943fa5e23099
I20260925 10:35:28.098273 15302 ts_tablet_manager.cc:1939] T f724920ba8fe454e8c68943fa5e23099 P dc3d44bdb2eb4acabbe96820ce4db1ed: Deleting consensus metadata
I20260925 10:35:28.099947 14797 catalog_manager.cc:4841] TS dc3d44bdb2eb4acabbe96820ce4db1ed (127.0.97.1:33633): tablet f724920ba8fe454e8c68943fa5e23099 (table test-table [id=75b0d21e33d94c3f8bf9042f23b62339]) successfully deleted
I20260925 10:35:28.101007 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:28.084510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.0.97.1:33633)
0925 10:35:28.084671 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:28.084678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:28.084776 (+ 98us) server_negotiation.cc:407] Connection header received
0925 10:35:28.084978 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:28.084986 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:28.085079 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:28.085183 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:28.086997 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:28.087808 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:28.089694 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:28.090295 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:28.090460 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:28.093670 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:28.093696 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:28.093712 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:28.093757 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:28.097739 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:28.098621 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:28.098630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:28.098634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:28.098795 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:28.099240 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:28.099245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:28.099249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:28.099760 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:28.100041 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:28.100427 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 10:35:28.100637 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:35:28.101603 14994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:28.104727 15304 ts_tablet_manager.cc:1918] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:35:28.104969 15304 log.cc:1196] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/f724920ba8fe454e8c68943fa5e23099
I20260925 10:35:28.105612 15304 ts_tablet_manager.cc:1939] T f724920ba8fe454e8c68943fa5e23099 P 6f4d65d8f3ef48eeae930d5b0531ffa1: Deleting consensus metadata
I20260925 10:35:28.107187 14799 catalog_manager.cc:4841] TS 6f4d65d8f3ef48eeae930d5b0531ffa1 (127.0.97.3:40949): tablet f724920ba8fe454e8c68943fa5e23099 (table test-table [id=75b0d21e33d94c3f8bf9042f23b62339]) successfully deleted
I20260925 10:35:28.195185 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 14850
I20260925 10:35:28.219607 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 15003
I20260925 10:35:28.246809 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:28.220451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55178 (local address 127.0.97.2:37349)
0925 10:35:28.220627 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:28.220632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:28.220650 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:35:28.220788 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:28.220791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:28.220853 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:28.220933 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:28.222680 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:28.223425 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:28.225356 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:28.225743 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:28.225832 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:28.240771 (+ 14939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:28.240787 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:28.240803 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:28.240847 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:28.243853 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:28.244674 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:28.244681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:28.244684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:28.244808 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:28.245183 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:28.245186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:28.245188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:28.245524 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:28.245786 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:28.246115 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 10:35:28.246275 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:35:28.247602 15126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:28.336252 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 15003
I20260925 10:35:28.359666 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 15134
I20260925 10:35:28.375626 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:28.360401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.0.97.3:40949)
0925 10:35:28.360613 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:28.360617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:28.360713 (+ 96us) server_negotiation.cc:407] Connection header received
0925 10:35:28.360899 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:28.360903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:28.360962 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:28.361043 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:28.362852 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:28.363671 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:28.365625 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:28.366103 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:28.366205 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:28.369569 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:28.369580 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:28.369595 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:28.369631 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:28.372667 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:28.373521 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:28.373528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:28.373530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:28.373667 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:28.374082 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:28.374085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:28.374087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:28.374489 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:28.374724 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:28.375065 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 10:35:28.375217 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:35:28.376359 15257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:28.496044 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 15134
I20260925 10:35:28.515985 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 14780
I20260925 10:35:28.517052 14843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:28.628100 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 14780
I20260925 10:35:28.649319 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43783
--webserver_interface=127.0.97.62
--webserver_port=36329
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:35:28.961687 15309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:28.961944 15309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:28.962095 15309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:28.968926 15309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:35:28.969010 15309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:28.969054 15309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:28.969134 15309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:35:28.969206 15309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:35:28.978386 15309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:28.979622 15309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:28.980858 15309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:28.991868 15317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:28.991906 15315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:28.992435 15314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:28.992822 15309 server_base.cc:1034] running on GCE node
I20260925 10:35:28.993464 15309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:28.994714 15309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:28.995981 15309 hybrid_clock.cc:648] HybridClock initialized: now 1790332528995921 us; error 43 us; skew 500 ppm
I20260925 10:35:28.996335 15309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:28 dist-test-slave-dxdd krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:28.999388 15309 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:35:29.000767 15309 webserver.cc:460] Webserver started at http://127.0.97.62:36329/ using document root <none> and password file <none>
I20260925 10:35:29.001368 15309 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:29.001499 15309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:29.006554 15309 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:35:29.009265 15324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:29.010519 15309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:35:29.010692 15309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "6b0d39c466504eef8ec3e7d55023612f"
format_stamp: "Formatted at 2026-09-25 10:35:23 on dist-test-slave-dxdd"
server_key: "b8c18ba86bea496b51b7e392d73a436d"
server_key_iv: "99e6e78019097f25ff7a19bf803323ee"
server_key_version: "encryptionkey@0"
I20260925 10:35:29.011641 15309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:29.027880 15309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:29.032073 15309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:29.032341 15309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:29.053153 15309 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43783
I20260925 10:35:29.053169 15375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43783 every 8 connection(s)
I20260925 10:35:29.055020 15309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:35:29.059365 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 15309
I20260925 10:35:29.060189 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33633
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43783
--webserver_port=34191
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:35:29.062276 15376 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:35:29.065392 15376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f: Bootstrap starting.
I20260925 10:35:29.085013 15376 log.cc:824] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:29.099869 15376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:35:29.100351 15376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f: Bootstrap complete.
I20260925 10:35:29.108793 15376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } }
I20260925 10:35:29.109599 15376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0d39c466504eef8ec3e7d55023612f, State: Initialized, Role: FOLLOWER
I20260925 10:35:29.110206 15376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } }
I20260925 10:35:29.110491 15376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:35:29.110603 15376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:35:29.110795 15376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:35:29.114398 15376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } }
I20260925 10:35:29.114907 15376 leader_election.cc:302] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b0d39c466504eef8ec3e7d55023612f; no voters:
I20260925 10:35:29.115613 15376 leader_election.cc:288] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 10:35:29.115821 15380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 3 FOLLOWER]: Leader election won for term 3
I20260925 10:35:29.117908 15376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:35:29.122949 15380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [term 3 LEADER]: Becoming Leader. State: Replica: 6b0d39c466504eef8ec3e7d55023612f, State: Running, Role: LEADER
I20260925 10:35:29.123493 15380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } }
I20260925 10:35:29.126171 15381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6b0d39c466504eef8ec3e7d55023612f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } } }
I20260925 10:35:29.126565 15381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:29.127235 15382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b0d39c466504eef8ec3e7d55023612f. Latest consensus state: current_term: 3 leader_uuid: "6b0d39c466504eef8ec3e7d55023612f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0d39c466504eef8ec3e7d55023612f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } } }
I20260925 10:35:29.127552 15382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0d39c466504eef8ec3e7d55023612f [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:29.131059 15392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:35:29.147786 15392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:35:29.149693 15392 catalog_manager.cc:1259] Loaded cluster ID: bb3956981dec43f3bb31901ee892836f
I20260925 10:35:29.149811 15392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 10:35:29.152350 15392 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f47a31ba9ad google::LogMessage::Fail() at ??:0
@ 0x7f47a31bd88a google::LogMessage::SendToLog() at ??:0
@ 0x7f47a31ba4de google::LogMessage::Flush() at ??:0
@ 0x7f47a31bba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f47b8bed512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f47b8bea976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f47b8c72610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f47b8c72439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f47a5e06345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f47a5e0991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f47a5e09799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f47a5de4eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f47a90be6db start_thread at ??:0
@ 0x7f47a205461f clone at ??:0
W20260925 10:35:29.411324 15378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:29.411635 15378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:29.411818 15378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:29.419139 15378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:29.419281 15378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:29.419445 15378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:35:29.428380 15378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34191
--tserver_master_addrs=127.0.97.62:43783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:29.429618 15378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:29.430959 15378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:29.443423 15418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:29.443404 15421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:29.443459 15419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:29.444214 15378 server_base.cc:1034] running on GCE node
I20260925 10:35:29.444774 15378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:29.445598 15378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:29.446810 15378 hybrid_clock.cc:648] HybridClock initialized: now 1790332529446764 us; error 70 us; skew 500 ppm
I20260925 10:35:29.447122 15378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:29 dist-test-slave-dxdd krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:29.450384 15378 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:35:29.451707 15378 webserver.cc:460] Webserver started at http://127.0.97.1:34191/ using document root <none> and password file <none>
I20260925 10:35:29.452337 15378 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:29.452467 15378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:29.457500 15378 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s
I20260925 10:35:29.460028 15428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:29.461244 15378 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:35:29.461412 15378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "dc3d44bdb2eb4acabbe96820ce4db1ed"
format_stamp: "Formatted at 2026-09-25 10:35:24 on dist-test-slave-dxdd"
server_key: "742019e08499c25a682dfa77ef966d88"
server_key_iv: "5df8b1cc2fd5197bdc21cd60a5ab5f20"
server_key_version: "encryptionkey@0"
I20260925 10:35:29.462363 15378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:29.483566 15378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:29.487623 15378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:29.487902 15378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:29.488777 15378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:29.490062 15378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:29.490202 15378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:29.490334 15378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:29.490433 15378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 10:35:29.522748 15378 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33633
I20260925 10:35:29.522826 15540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33633 every 8 connection(s)
I20260925 10:35:29.524119 15378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:35:29.528493 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:29.526836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:60657)
0925 10:35:29.527513 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:29.527700 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":259,"threads_started":1}
W20260925 10:35:29.529055 15541 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43783 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:43783: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
I20260925 10:35:29.529747 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:29.529475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:47683)
0925 10:35:29.529595 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:29.529653 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 10:35:29.530567 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:29.530219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:57079)
0925 10:35:29.530378 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:29.530468 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 10:35:29.531159 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 15378
I20260925 10:35:29.531706 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37349
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43783
--webserver_port=36013
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:35:29.534504 15541 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:35:29.846845 15544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:29.847088 15544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:29.847268 15544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:29.854384 15544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:29.854514 15544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:29.854701 15544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:35:29.863965 15544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36013
--tserver_master_addrs=127.0.97.62:43783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:29.865259 15544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:29.866626 15544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:29.879140 15549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:29.879072 15550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:29.879105 15552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:29.879957 15544 server_base.cc:1034] running on GCE node
I20260925 10:35:29.880546 15544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:29.881428 15544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:29.882613 15544 hybrid_clock.cc:648] HybridClock initialized: now 1790332529882581 us; error 29 us; skew 500 ppm
I20260925 10:35:29.882911 15544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:29 dist-test-slave-dxdd krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:29.885869 15544 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:35:29.887276 15544 webserver.cc:460] Webserver started at http://127.0.97.2:36013/ using document root <none> and password file <none>
I20260925 10:35:29.887949 15544 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:29.888077 15544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:29.892988 15544 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 10:35:29.897411 15559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:29.898742 15544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:35:29.898916 15544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "7e146c068db545d596246e5e6cbf7fd6"
format_stamp: "Formatted at 2026-09-25 10:35:24 on dist-test-slave-dxdd"
server_key: "4aadb17a7a3e107a604fdcf784b6e10e"
server_key_iv: "a9364526c51d3b8c1263d31b37d132ec"
server_key_version: "encryptionkey@0"
I20260925 10:35:29.899844 15544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:29.914095 15544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:29.918218 15544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:29.918490 15544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:29.919353 15544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:29.920508 15544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:29.920624 15544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:29.920755 15544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:29.920848 15544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:29.952452 15544 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37349
I20260925 10:35:29.952514 15671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37349 every 8 connection(s)
I20260925 10:35:29.953820 15544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:35:29.958166 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:29.956494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:53953)
0925 10:35:29.957213 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:29.957396 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":269,"threads_started":1}
W20260925 10:35:29.958855 15672 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43783 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:43783: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
I20260925 10:35:29.959558 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:29.959285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:44259)
0925 10:35:29.959413 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:29.959470 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 10:35:29.960292 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:29.960026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:47619)
0925 10:35:29.960128 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:29.960191 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 10:35:29.960507 15672 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:35:29.962159 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 15544
I20260925 10:35:29.962721 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40949
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43783
--webserver_port=39733
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:35:30.267943 15675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:30.268214 15675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:30.268393 15675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:30.275511 15675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:30.275624 15675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:30.275846 15675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:35:30.284984 15675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39733
--tserver_master_addrs=127.0.97.62:43783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:30.286118 15675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:30.287403 15675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:30.299815 15680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:30.299818 15683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:30.299840 15681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:30.300508 15675 server_base.cc:1034] running on GCE node
I20260925 10:35:30.301023 15675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:30.301865 15675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:30.303097 15675 hybrid_clock.cc:648] HybridClock initialized: now 1790332530303045 us; error 43 us; skew 500 ppm
I20260925 10:35:30.303397 15675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:30 dist-test-slave-dxdd krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:30.306362 15675 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:35:30.307616 15675 webserver.cc:460] Webserver started at http://127.0.97.3:39733/ using document root <none> and password file <none>
I20260925 10:35:30.308252 15675 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:30.308388 15675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:30.313186 15675 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 10:35:30.315676 15690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:30.316978 15675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:35:30.317135 15675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "6f4d65d8f3ef48eeae930d5b0531ffa1"
format_stamp: "Formatted at 2026-09-25 10:35:25 on dist-test-slave-dxdd"
server_key: "95e9a4c48d4dec08c6b290347a2e710e"
server_key_iv: "7520057a35ce30d2dd607b0ef929c575"
server_key_version: "encryptionkey@0"
I20260925 10:35:30.318043 15675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:30.336215 15675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:30.340106 15675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:30.340389 15675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:30.341293 15675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:30.342598 15675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:30.342703 15675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:30.342831 15675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:30.342921 15675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:30.372709 15675 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40949
I20260925 10:35:30.372761 15802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40949 every 8 connection(s)
I20260925 10:35:30.374080 15675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:35:30.378567 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:30.376861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:41443)
0925 10:35:30.377536 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:30.377713 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":265,"threads_started":1}
W20260925 10:35:30.379153 15803 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43783 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:43783: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
I20260925 10:35:30.379928 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:30.379631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:52745)
0925 10:35:30.379769 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:30.379837 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 10:35:30.380704 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:30.380381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:33397)
0925 10:35:30.380515 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:30.380590 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 10:35:30.380923 15803 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:35:30.382649 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 15675
I20260925 10:35:30.535790 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:30.535017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:45333)
0925 10:35:30.535479 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:30.535565 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":233,"threads_started":1}
I20260925 10:35:30.961982 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:30.961088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:56543)
0925 10:35:30.961533 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:30.961659 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":204,"threads_started":1}
I20260925 10:35:31.382226 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:31.381479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:39909)
0925 10:35:31.381936 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:31.382018 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":253,"threads_started":1}
I20260925 10:35:31.537278 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:31.536619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:54795)
0925 10:35:31.537048 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:31.537120 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":226,"threads_started":1}
I20260925 10:35:31.963465 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:31.962785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:33523)
0925 10:35:31.963228 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:31.963311 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":233,"threads_started":1}
I20260925 10:35:32.383714 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:32.383064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:40723)
0925 10:35:32.383480 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:32.383565 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":208,"threads_started":1}
I20260925 10:35:32.538842 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:32.538149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:33283)
0925 10:35:32.538594 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:32.538664 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":256,"threads_started":1}
I20260925 10:35:32.965054 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:32.964368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:47781)
0925 10:35:32.964822 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:32.964896 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":212,"threads_started":1}
I20260925 10:35:33.385486 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:33.384744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:33075)
0925 10:35:33.385264 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:33.385333 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":302,"threads_started":1}
I20260925 10:35:33.540293 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:33.539630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:40333)
0925 10:35:33.540037 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:33.540111 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":204,"threads_started":1}
I20260925 10:35:33.966701 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:33.965900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:34267)
0925 10:35:33.966448 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:33.966522 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":285,"threads_started":1}
I20260925 10:35:34.386970 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:34.386273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:52411)
0925 10:35:34.386729 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:34.386797 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":214,"threads_started":1}
I20260925 10:35:34.541893 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:34.541157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:55793)
0925 10:35:34.541632 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:34.541725 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":259,"threads_started":1}
I20260925 10:35:34.968191 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:34.967464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:58983)
0925 10:35:34.967966 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:34.968036 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":297,"threads_started":1}
I20260925 10:35:35.388440 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:35.387778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:39707)
0925 10:35:35.388210 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:35.388284 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":231,"threads_started":1}
I20260925 10:35:35.543766 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:35.542956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:33101)
0925 10:35:35.543478 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:35.543584 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":304,"threads_started":1}
I20260925 10:35:35.969691 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:35.969040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:46137)
0925 10:35:35.969459 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:35.969539 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":226,"threads_started":1}
I20260925 10:35:36.390000 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:36.389350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:48377)
0925 10:35:36.389761 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:36.389843 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":220,"threads_started":1}
I20260925 10:35:36.545575 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:36.544832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:43439)
0925 10:35:36.545285 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:36.545358 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":212,"threads_started":1}
I20260925 10:35:36.971343 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:36.970560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:42121)
0925 10:35:36.971022 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:36.971098 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":207,"threads_started":1}
I20260925 10:35:37.391626 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:37.390883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:45793)
0925 10:35:37.391330 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:37.391405 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":216,"threads_started":1}
I20260925 10:35:37.547186 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:37.546548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:43025)
0925 10:35:37.546948 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:37.547020 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":204,"threads_started":1}
I20260925 10:35:37.973037 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:37.972366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:60951)
0925 10:35:37.972808 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:37.972884 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":234,"threads_started":1}
I20260925 10:35:38.393074 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:38.392419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:53965)
0925 10:35:38.392849 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:38.392921 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":229,"threads_started":1}
I20260925 10:35:38.548826 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:38.548076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:47541)
0925 10:35:38.548576 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:38.548658 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":243,"threads_started":1}
I20260925 10:35:38.974696 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:38.973843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:43363)
0925 10:35:38.974447 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:38.974528 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":323,"threads_started":1}
I20260925 10:35:39.394773 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:39.393960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:50531)
0925 10:35:39.394472 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:39.394580 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":210,"threads_started":1}
I20260925 10:35:39.550494 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:39.549683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:42351)
0925 10:35:39.550190 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:39.550263 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":305,"threads_started":1}
I20260925 10:35:39.976245 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:39.975550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:33917)
0925 10:35:39.975982 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:39.976057 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":236,"threads_started":1}
I20260925 10:35:40.396271 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:40.395640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:54971)
0925 10:35:40.396050 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:40.396125 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":233,"threads_started":1}
I20260925 10:35:40.552059 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:40.551389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:34581)
0925 10:35:40.551788 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:40.551884 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":203,"threads_started":1}
I20260925 10:35:40.977846 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:40.977191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:44233)
0925 10:35:40.977583 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:40.977677 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":190,"threads_started":1}
I20260925 10:35:41.397751 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:41.397120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:39527)
0925 10:35:41.397507 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:41.397588 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":196,"threads_started":1}
I20260925 10:35:41.553572 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:41.552910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:56427)
0925 10:35:41.553332 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:41.553418 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":211,"threads_started":1}
I20260925 10:35:41.979383 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:41.978709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:39819)
0925 10:35:41.979148 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:41.979225 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":235,"threads_started":1}
I20260925 10:35:42.399312 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:42.398707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:46965)
0925 10:35:42.399096 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:42.399169 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":200,"threads_started":1}
I20260925 10:35:42.555145 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:42.554448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:57987)
0925 10:35:42.554884 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:42.554964 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":207,"threads_started":1}
I20260925 10:35:42.980919 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:42.980219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:34661)
0925 10:35:42.980659 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:42.980739 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":202,"threads_started":1}
I20260925 10:35:43.400868 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:43.400174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:49287)
0925 10:35:43.400611 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:43.400688 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":207,"threads_started":1}
I20260925 10:35:43.556674 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:43.556013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:50319)
0925 10:35:43.556428 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:43.556503 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":212,"threads_started":1}
I20260925 10:35:43.982645 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:43.981865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:50271)
0925 10:35:43.982375 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:43.982468 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":257,"threads_started":1}
I20260925 10:35:44.402407 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:44.401679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:43439)
0925 10:35:44.402094 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:44.402168 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":222,"threads_started":1}
I20260925 10:35:44.558432 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:44.557686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.1:59517)
0925 10:35:44.558108 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:44.558182 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":227,"threads_started":1}
I20260925 10:35:44.984351 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:44.983634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.2:40097)
0925 10:35:44.984095 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:44.984172 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":237,"threads_started":1}
I20260925 10:35:45.387351 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 15378
I20260925 10:35:45.404338 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:45.403500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43783 (local address 127.0.97.3:55437)
0925 10:35:45.403980 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:45.404070 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":239,"threads_started":1}
I20260925 10:35:45.405571 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:45.388670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.0.97.1:33633)
0925 10:35:45.389340 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:45.389361 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:45.389408 (+ 47us) server_negotiation.cc:407] Connection header received
0925 10:35:45.391194 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:45.391204 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:45.391636 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:45.391998 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:45.393425 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:45.394739 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:45.396003 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:45.396660 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:45.396867 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:45.399472 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:45.399480 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:45.399494 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:45.399529 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:45.402158 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:45.402870 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:45.402880 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:45.402893 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:45.403085 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:45.403517 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:45.403521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:45.403523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:45.404281 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:45.404611 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:45.404987 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 10:35:45.405308 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":509,"threads_started":1}
I20260925 10:35:45.406983 15536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:45.490149 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 15378
I20260925 10:35:45.506779 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 15544
I20260925 10:35:45.524458 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:45.507686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.0.97.2:37349)
0925 10:35:45.508168 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:45.508181 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:45.508217 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:35:45.509704 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:45.509713 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:45.510166 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:45.510563 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:45.511975 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:45.513169 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:45.514448 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:45.515155 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:45.515362 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:45.518295 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:45.518344 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:45.518364 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:45.518420 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:45.521068 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:45.521723 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:45.521732 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:45.521740 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:45.521977 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:45.522496 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:45.522501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:45.522504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:45.523017 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:45.523328 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:45.523766 (+ 438us) server_negotiation.cc:299] Negotiation successful
0925 10:35:45.524177 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":227,"threads_started":1}
I20260925 10:35:45.525806 15667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:45.609452 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 15544
I20260925 10:35:45.626657 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 15675
I20260925 10:35:45.644301 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:45.627662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.0.97.3:40949)
0925 10:35:45.628228 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:45.628242 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:45.628280 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:35:45.629918 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:45.629927 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:45.630454 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:45.630904 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:45.632372 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:45.633581 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:45.634910 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:45.635587 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:45.635792 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:45.638429 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:45.638437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:45.638452 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:45.638491 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:45.641096 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:45.641764 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:45.641772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:45.641781 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:45.641943 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:45.642339 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:45.642345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:45.642348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:45.642849 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:45.643193 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:45.643596 (+ 403us) server_negotiation.cc:299] Negotiation successful
0925 10:35:45.644003 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":268,"threads_started":1}
I20260925 10:35:45.645627 15798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:45.729221 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 15675
2026-09-25T10:35:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23785 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:35:45 dist-test-slave-dxdd krb5kdc[15860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:35:45 dist-test-slave-dxdd krb5kdc[15860](info): set up 2 sockets
Sep 25 10:35:45 dist-test-slave-dxdd krb5kdc[15860](info): commencing operation
krb5kdc: starting...
W20260925 10:35:46.827008 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:35:46 dist-test-slave-dxdd krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:35:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:35:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:35:47.070003 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46815
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:35:47.392778 15873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:47.393079 15873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:47.393168 15873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:47.401532 15873 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:35:47.401631 15873 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:35:47.401669 15873 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:35:47.401721 15873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:35:47.401775 15873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:47.401842 15873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:47.401886 15873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:35:47.401940 15873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:35:47.411711 15873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:47.412925 15873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:47.414143 15873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:47.425631 15878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:47.425721 15879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:47.425680 15881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:47.426578 15873 server_base.cc:1034] running on GCE node
I20260925 10:35:47.427237 15873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:47.428496 15873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:47.429709 15873 hybrid_clock.cc:648] HybridClock initialized: now 1790332547429653 us; error 28 us; skew 500 ppm
I20260925 10:35:47.429986 15873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:47 dist-test-slave-dxdd krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:47.433128 15873 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:35:47.434613 15873 webserver.cc:460] Webserver started at http://127.0.97.62:45773/ using document root <none> and password file <none>
I20260925 10:35:47.435257 15873 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:47.435343 15873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:47.435601 15873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:35:47.438423 15873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "98adecbb715843819b43c16086209ed1"
format_stamp: "Formatted at 2026-09-25 10:35:47 on dist-test-slave-dxdd"
server_key: "a1f396eb17d4bf33e2f3cd4e28a869d6"
server_key_iv: "536d95c5091d0755fbcc2853eaa197cd"
server_key_version: "encryptionkey@0"
I20260925 10:35:47.439126 15873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "98adecbb715843819b43c16086209ed1"
format_stamp: "Formatted at 2026-09-25 10:35:47 on dist-test-slave-dxdd"
server_key: "a1f396eb17d4bf33e2f3cd4e28a869d6"
server_key_iv: "536d95c5091d0755fbcc2853eaa197cd"
server_key_version: "encryptionkey@0"
I20260925 10:35:47.443852 15873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:35:47.447072 15888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:47.448308 15873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:35:47.448441 15873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "98adecbb715843819b43c16086209ed1"
format_stamp: "Formatted at 2026-09-25 10:35:47 on dist-test-slave-dxdd"
server_key: "a1f396eb17d4bf33e2f3cd4e28a869d6"
server_key_iv: "536d95c5091d0755fbcc2853eaa197cd"
server_key_version: "encryptionkey@0"
I20260925 10:35:47.448611 15873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:47.460285 15873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:47.464339 15873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:47.464547 15873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:47.483170 15873 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46815
I20260925 10:35:47.483189 15939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46815 every 8 connection(s)
I20260925 10:35:47.484513 15873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:35:47.488080 15940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:47.492360 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 15873
I20260925 10:35:47.492575 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:35:47.493410 388 external_mini_cluster.cc:1428] Setting key 8bd9bcc13dfe9519c8d9e764028243fc
I20260925 10:35:47.499627 15940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Bootstrap starting.
I20260925 10:35:47.503201 15940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:47.504424 15940 log.cc:824] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:47.507881 15940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: No bootstrap required, opened a new log
Sep 25 10:35:47 dist-test-slave-dxdd krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:47.516083 15940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } }
I20260925 10:35:47.516498 15940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:47.516636 15940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98adecbb715843819b43c16086209ed1, State: Initialized, Role: FOLLOWER
I20260925 10:35:47.517309 15943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:47.495443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.0.97.62:46815)
0925 10:35:47.496208 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:47.496219 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:47.496258 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:35:47.498121 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:47.498144 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:47.498623 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:47.498959 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:47.500710 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:47.502015 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:47.504060 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:47.504780 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:47.505010 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:47.509905 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:47.509913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:47.509930 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:47.509969 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:47.513104 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:47.513835 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:47.513853 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:47.513859 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:47.514024 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:47.514448 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:47.514455 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:47.514462 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:47.514960 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:47.515332 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:47.515690 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 10:35:47.516109 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":358,"threads_started":1}
I20260925 10:35:47.517217 15940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } }
I20260925 10:35:47.517493 15940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:35:47.517664 15940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:35:47.517853 15940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:47.520288 15940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } }
I20260925 10:35:47.520768 15940 leader_election.cc:302] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98adecbb715843819b43c16086209ed1; no voters:
I20260925 10:35:47.521361 15940 leader_election.cc:288] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:35:47.521516 15945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:35:47.522214 15945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 1 LEADER]: Becoming Leader. State: Replica: 98adecbb715843819b43c16086209ed1, State: Running, Role: LEADER
I20260925 10:35:47.522818 15945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } }
I20260925 10:35:47.523386 15940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:35:47.525749 15947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98adecbb715843819b43c16086209ed1. Latest consensus state: current_term: 1 leader_uuid: "98adecbb715843819b43c16086209ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } } }
I20260925 10:35:47.526125 15947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:47.526347 15946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98adecbb715843819b43c16086209ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } } }
I20260925 10:35:47.526705 15946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:35:47.528630 15951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:35:47.533560 15951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:35:47.542438 15951 catalog_manager.cc:1347] Generated new cluster ID: bf100fd3ddad4a0f940805b4068fbbc7
I20260925 10:35:47.542542 15951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:35:47.561860 15951 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:35:47.563208 15951 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:35:47.570328 15951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Generated new TSK 0
I20260925 10:35:47.571112 15951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:35:47.643389 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46815
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:35:47.967579 15966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:47.967839 15966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:47.968024 15966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:47.975172 15966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:47.975266 15966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:47.975476 15966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:35:47.985436 15966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:47.986694 15966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:47.987986 15966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:48.001853 15971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:48.001886 15972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:48.002194 15974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:48.002667 15966 server_base.cc:1034] running on GCE node
I20260925 10:35:48.003196 15966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:48.004086 15966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:48.005280 15966 hybrid_clock.cc:648] HybridClock initialized: now 1790332548005235 us; error 33 us; skew 500 ppm
I20260925 10:35:48.005602 15966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:48 dist-test-slave-dxdd krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:48.008757 15966 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:35:48.010129 15966 webserver.cc:460] Webserver started at http://127.0.97.1:36669/ using document root <none> and password file <none>
I20260925 10:35:48.010846 15966 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:48.010977 15966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:48.011294 15966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:35:48.014366 15966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "76a782401dd049179383645f3588efcb"
format_stamp: "Formatted at 2026-09-25 10:35:48 on dist-test-slave-dxdd"
server_key: "da5f38e002ef3b13ee160f2372643af7"
server_key_iv: "f0e73c0cb771092015ac8869e003e07b"
server_key_version: "encryptionkey@0"
I20260925 10:35:48.015106 15966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "76a782401dd049179383645f3588efcb"
format_stamp: "Formatted at 2026-09-25 10:35:48 on dist-test-slave-dxdd"
server_key: "da5f38e002ef3b13ee160f2372643af7"
server_key_iv: "f0e73c0cb771092015ac8869e003e07b"
server_key_version: "encryptionkey@0"
I20260925 10:35:48.019980 15966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:35:48.023312 15981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:48.024770 15966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:35:48.024962 15966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "76a782401dd049179383645f3588efcb"
format_stamp: "Formatted at 2026-09-25 10:35:48 on dist-test-slave-dxdd"
server_key: "da5f38e002ef3b13ee160f2372643af7"
server_key_iv: "f0e73c0cb771092015ac8869e003e07b"
server_key_version: "encryptionkey@0"
I20260925 10:35:48.025184 15966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:48.045524 15966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:48.049376 15966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:48.049657 15966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:48.050585 15966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:48.051843 15966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:48.051962 15966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:48.052119 15966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:48.052192 15966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:48.084923 15966 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41979
I20260925 10:35:48.084968 16093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41979 every 8 connection(s)
I20260925 10:35:48.086292 15966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:35:48.095610 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 15966
I20260925 10:35:48.095829 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:35:48.096477 388 external_mini_cluster.cc:1428] Setting key f07512ca28c51139c43c2509584e10dd
Sep 25 10:35:48 dist-test-slave-dxdd krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:48.105799 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:48.088789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34585 (local address 127.0.97.62:46815)
0925 10:35:48.089194 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:48.089199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:48.090215 (+ 1016us) server_negotiation.cc:407] Connection header received
0925 10:35:48.092104 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:48.092109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:48.092199 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:48.092327 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:48.093742 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:48.094508 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:48.096165 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:48.096553 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:48.096684 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:48.100389 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:48.100394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:48.100397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:48.100426 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:48.102936 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:48.103610 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:48.103617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:48.103620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:48.103744 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:48.104168 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:48.104174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:48.104178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:48.104425 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:48.104585 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:48.105229 (+ 644us) server_negotiation.cc:299] Negotiation successful
0925 10:35:48.105438 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":198,"threads_started":1}
I20260925 10:35:48.106802 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:48.089374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:34585)
0925 10:35:48.090062 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:48.090098 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:48.091831 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:48.092555 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:48.092562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:48.093029 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:48.093567 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:48.093583 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:48.094667 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:48.094677 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:48.096015 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:48.096027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:48.096798 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:48.096804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:48.097090 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:48.097802 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:48.097827 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:48.100174 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:48.103083 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:48.103090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:48.103105 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:48.103495 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:48.103852 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:48.103857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:48.103860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:48.104056 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:48.104548 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:48.104556 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:48.104866 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 10:35:48.105134 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 10:35:48.105630 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":275,"threads_started":1}
I20260925 10:35:48.108292 16094 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46815
I20260925 10:35:48.108597 16094 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:48.109407 16094 heartbeater.cc:511] Master 127.0.97.62:46815 requested a full tablet report, sending...
I20260925 10:35:48.111471 15905 ts_manager.cc:194] Registered new tserver with Master: 76a782401dd049179383645f3588efcb (127.0.97.1:41979)
I20260925 10:35:48.112799 15905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34585
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:35:48.147394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46815
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:35:48.465967 16100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:48.466267 16100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:48.466454 16100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:48.473635 16100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:48.473738 16100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:48.473851 16100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:35:48.483296 16100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:48.484503 16100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:48.485762 16100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:48.500325 16108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:48.500319 16105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:48.500252 16106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:48.501539 16100 server_base.cc:1034] running on GCE node
I20260925 10:35:48.502189 16100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:48.503181 16100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:48.504372 16100 hybrid_clock.cc:648] HybridClock initialized: now 1790332548504313 us; error 60 us; skew 500 ppm
I20260925 10:35:48.504676 16100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:48 dist-test-slave-dxdd krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:48.513134 16100 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:35:48.514532 16100 webserver.cc:460] Webserver started at http://127.0.97.2:34343/ using document root <none> and password file <none>
I20260925 10:35:48.515153 16100 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:48.515246 16100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:48.515493 16100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:35:48.518211 16100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "6b469cc748964e6197a3734599e37e1b"
format_stamp: "Formatted at 2026-09-25 10:35:48 on dist-test-slave-dxdd"
server_key: "930e9d6734836d25eca214d583bdd6f7"
server_key_iv: "f09828dc763dec5eef95751a1488fd80"
server_key_version: "encryptionkey@0"
I20260925 10:35:48.518971 16100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6b469cc748964e6197a3734599e37e1b"
format_stamp: "Formatted at 2026-09-25 10:35:48 on dist-test-slave-dxdd"
server_key: "930e9d6734836d25eca214d583bdd6f7"
server_key_iv: "f09828dc763dec5eef95751a1488fd80"
server_key_version: "encryptionkey@0"
I20260925 10:35:48.523697 16100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 10:35:48.527032 16115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:48.528353 16100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:35:48.528470 16100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "6b469cc748964e6197a3734599e37e1b"
format_stamp: "Formatted at 2026-09-25 10:35:48 on dist-test-slave-dxdd"
server_key: "930e9d6734836d25eca214d583bdd6f7"
server_key_iv: "f09828dc763dec5eef95751a1488fd80"
server_key_version: "encryptionkey@0"
I20260925 10:35:48.528635 16100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:48.551378 16100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:48.555122 16100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:48.555341 16100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:48.556142 16100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:48.557284 16100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:48.557358 16100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:48.557436 16100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:48.557524 16100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:48.590194 16100 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45481
I20260925 10:35:48.590241 16227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45481 every 8 connection(s)
I20260925 10:35:48.591456 16100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:35:48.597862 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 16100
I20260925 10:35:48.598052 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:35:48.598551 388 external_mini_cluster.cc:1428] Setting key b924b74d1ea9470fc6883effa997fcdd
Sep 25 10:35:48 dist-test-slave-dxdd krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:48.612551 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:48.593945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43761 (local address 127.0.97.62:46815)
0925 10:35:48.594120 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:48.594124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:48.595311 (+ 1187us) server_negotiation.cc:407] Connection header received
0925 10:35:48.597274 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:48.597288 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:48.597399 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:48.597507 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:48.599240 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:48.600052 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:48.601645 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:48.602449 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:48.602589 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:48.605782 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:48.605788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:48.605795 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:48.605840 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:48.609449 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:48.610366 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:48.610371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:48.610375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:48.610510 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:48.610967 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:48.610970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:48.610972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:48.611214 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:48.611315 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:48.612161 (+ 846us) server_negotiation.cc:299] Negotiation successful
0925 10:35:48.612319 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:35:48.614054 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:48.594459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:43761)
0925 10:35:48.595135 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:48.595176 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:48.597009 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:48.597746 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:48.597753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:48.598415 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:48.599037 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:48.599064 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:48.600239 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:48.600244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:48.601498 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:48.601510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:48.602635 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:48.602640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:48.602822 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:48.603487 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:48.603508 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:48.605557 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:48.609582 (+ 4025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:48.609594 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:48.609612 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:48.610214 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:48.610618 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:48.610625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:48.610632 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:48.610860 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:48.611371 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:48.611381 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:48.611789 (+ 408us) client_negotiation.cc:769] Sending connection context
0925 10:35:48.612087 (+ 298us) client_negotiation.cc:240] Negotiation successful
0925 10:35:48.612623 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":276,"threads_started":1}
I20260925 10:35:48.615506 16228 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46815
I20260925 10:35:48.615828 16228 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:48.616498 16228 heartbeater.cc:511] Master 127.0.97.62:46815 requested a full tablet report, sending...
I20260925 10:35:48.617901 15905 ts_manager.cc:194] Registered new tserver with Master: 6b469cc748964e6197a3734599e37e1b (127.0.97.2:45481)
I20260925 10:35:48.618853 15905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43761
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:35:48.650028 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46815
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:35:48.960305 16233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:35:48.960587 16233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:35:48.960747 16233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:35:48.967614 16233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:35:48.967739 16233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:35:48.967926 16233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:35:48.977185 16233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:35:48.978472 16233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:35:48.979725 16233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:35:48.992115 16239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:48.992082 16238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:35:48.992163 16241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:35:48.992831 16233 server_base.cc:1034] running on GCE node
I20260925 10:35:48.993364 16233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:35:48.994203 16233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:35:48.995396 16233 hybrid_clock.cc:648] HybridClock initialized: now 1790332548995351 us; error 37 us; skew 500 ppm
I20260925 10:35:48.995645 16233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:35:48 dist-test-slave-dxdd krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:35:48.998504 16233 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:35:48.999826 16233 webserver.cc:460] Webserver started at http://127.0.97.3:45451/ using document root <none> and password file <none>
I20260925 10:35:49.000451 16233 fs_manager.cc:359] Metadata directory not provided
I20260925 10:35:49.000581 16233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:35:49.000886 16233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:35:49.003618 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "c5dc926b46da4f3db8c4f1e66852ad14"
format_stamp: "Formatted at 2026-09-25 10:35:48 on dist-test-slave-dxdd"
server_key: "d85ada91ed0ae8cc86fb630e4f1c60da"
server_key_iv: "02fba15dbb21f7876350ed95d698f01f"
server_key_version: "encryptionkey@0"
I20260925 10:35:49.004346 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c5dc926b46da4f3db8c4f1e66852ad14"
format_stamp: "Formatted at 2026-09-25 10:35:48 on dist-test-slave-dxdd"
server_key: "d85ada91ed0ae8cc86fb630e4f1c60da"
server_key_iv: "02fba15dbb21f7876350ed95d698f01f"
server_key_version: "encryptionkey@0"
I20260925 10:35:49.009169 16233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 10:35:49.012465 16248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:49.013681 16233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:35:49.013849 16233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "c5dc926b46da4f3db8c4f1e66852ad14"
format_stamp: "Formatted at 2026-09-25 10:35:48 on dist-test-slave-dxdd"
server_key: "d85ada91ed0ae8cc86fb630e4f1c60da"
server_key_iv: "02fba15dbb21f7876350ed95d698f01f"
server_key_version: "encryptionkey@0"
I20260925 10:35:49.014034 16233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:35:49.029292 16233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:35:49.033473 16233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:35:49.033766 16233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:35:49.034813 16233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:35:49.036041 16233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:35:49.036119 16233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:49.036199 16233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:35:49.036240 16233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:35:49.072364 16233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34189
I20260925 10:35:49.072407 16360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34189 every 8 connection(s)
I20260925 10:35:49.073623 16233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:35:49.079473 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 16233
I20260925 10:35:49.079656 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:35:49.080278 388 external_mini_cluster.cc:1428] Setting key f270f0bbc720c2e6acd1492465364af0
Sep 25 10:35:49 dist-test-slave-dxdd krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:35:49.093405 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.076087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37445 (local address 127.0.97.62:46815)
0925 10:35:49.076299 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:49.076305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:49.077317 (+ 1012us) server_negotiation.cc:407] Connection header received
0925 10:35:49.079091 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:49.079095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:49.079160 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:49.079263 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:49.081306 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.082298 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.083866 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.084245 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.084378 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.087946 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:49.087950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:49.087954 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:49.087987 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:49.090567 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:49.091235 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:49.091242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:49.091246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:49.091373 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:49.091778 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:49.091785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:49.091788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:49.092040 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:49.092191 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:49.092824 (+ 633us) server_negotiation.cc:299] Negotiation successful
0925 10:35:49.093044 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 10:35:49.094465 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.076498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:37445)
0925 10:35:49.077165 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:49.077201 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:49.078851 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:49.079530 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:49.079555 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:49.080262 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:35:49.081126 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:49.081145 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.082504 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.082515 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:49.083710 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:49.083722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.084467 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.084477 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:49.084719 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.085381 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:35:49.085402 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:35:49.087749 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:35:49.090696 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:49.090704 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:49.090718 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:49.091116 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:49.091484 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:35:49.091489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:35:49.091493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:35:49.091666 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:35:49.092163 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:35:49.092170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:35:49.092465 (+ 295us) client_negotiation.cc:769] Sending connection context
0925 10:35:49.092728 (+ 263us) client_negotiation.cc:240] Negotiation successful
0925 10:35:49.093197 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":264,"threads_started":1}
I20260925 10:35:49.095860 16361 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46815
I20260925 10:35:49.096179 16361 heartbeater.cc:464] Registering TS with master...
I20260925 10:35:49.096832 16361 heartbeater.cc:511] Master 127.0.97.62:46815 requested a full tablet report, sending...
I20260925 10:35:49.098098 15905 ts_manager.cc:194] Registered new tserver with Master: c5dc926b46da4f3db8c4f1e66852ad14 (127.0.97.3:34189)
I20260925 10:35:49.098904 15905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37445
I20260925 10:35:49.107476 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:35:49.116319 16094 heartbeater.cc:503] Master 127.0.97.62:46815 was elected leader, sending a full tablet report...
I20260925 10:35:49.124085 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.111681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48356 (local address 127.0.97.62:46815)
0925 10:35:49.111842 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:49.111846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:49.112253 (+ 407us) server_negotiation.cc:407] Connection header received
0925 10:35:49.112382 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:49.112386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:49.112438 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:49.112513 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:49.114053 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.114806 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.116190 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.116737 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.116880 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.119257 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:49.119262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:49.119265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:49.119291 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:49.121824 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:49.122430 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:49.122433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:49.122435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:49.122550 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:49.122846 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:49.122849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:49.122850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:49.123122 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:49.123295 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:49.123672 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 10:35:49.123825 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:35:49.127592 15905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:35:49.134495 15905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:35:49.156330 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.145703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.0.97.3:34189)
0925 10:35:49.146480 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:49.146490 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:49.147315 (+ 825us) server_negotiation.cc:407] Connection header received
0925 10:35:49.147711 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:49.147718 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:49.147974 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:49.148144 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:49.149380 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.150963 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.153893 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.155115 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.155188 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.155342 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:49.155819 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 10:35:49.155975 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":432,"threads_started":1}
I20260925 10:35:49.156376 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.145562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34189 (local address 127.0.0.1:47650)
0925 10:35:49.147174 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:49.147218 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:49.147449 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:49.148525 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:49.148533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:49.148572 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:49.149204 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:49.149216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.151178 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.151183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:49.153720 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:49.153734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.155478 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.155482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:49.155607 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.155637 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 10:35:49.155780 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 10:35:49.155959 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1399,"mutex_wait_us":25,"thread_start_us":2901,"threads_started":1}
I20260925 10:35:49.157621 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.145977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41022 (local address 127.0.97.2:45481)
0925 10:35:49.148316 (+ 2339us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:49.148324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:49.148852 (+ 528us) server_negotiation.cc:407] Connection header received
0925 10:35:49.149155 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:49.149164 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:49.149391 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:49.149554 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:49.150408 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.151936 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.154715 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.156132 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.156255 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.156435 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:49.157129 (+ 694us) server_negotiation.cc:299] Negotiation successful
0925 10:35:49.157290 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2177,"thread_start_us":2013,"threads_started":1}
I20260925 10:35:49.157750 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.147056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41979 (local address 127.0.0.1:37034)
0925 10:35:49.149555 (+ 2499us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:49.149575 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:49.149685 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:49.150398 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:49.150403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:49.150427 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:49.150851 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:49.150865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.152428 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.152438 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:49.154490 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:49.154504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.156656 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.156663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:49.156881 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.156964 (+ 83us) client_negotiation.cc:769] Sending connection context
0925 10:35:49.157056 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 10:35:49.157228 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2355,"mutex_wait_us":209,"thread_start_us":1840,"threads_started":1}
I20260925 10:35:49.158200 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.145577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45481 (local address 127.0.0.1:41022)
0925 10:35:49.148740 (+ 3163us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:49.148763 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:49.148949 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:49.149819 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:49.149823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:49.149850 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:49.150179 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:49.150191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.152109 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.152114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:49.154499 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:49.154521 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.156617 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.156626 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:49.156866 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.156959 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 10:35:49.157046 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:35:49.157228 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3018,"mutex_wait_us":243,"thread_start_us":1257,"threads_started":1}
I20260925 10:35:49.158576 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.146705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37034 (local address 127.0.97.1:41979)
0925 10:35:49.149280 (+ 2575us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:49.149309 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:49.149672 (+ 363us) server_negotiation.cc:407] Connection header received
0925 10:35:49.149864 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:49.149870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:49.150018 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:49.150197 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:49.151129 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.152294 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.154690 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.156132 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.156255 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.156465 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:49.158070 (+ 1605us) server_negotiation.cc:299] Negotiation successful
0925 10:35:49.158229 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2418,"thread_start_us":2327,"threads_started":1}
I20260925 10:35:49.160175 16296 tablet_service.cc:1467] Processing CreateTablet for tablet 21b6afd81fe949f6a9e77b0951a4ee34 (DEFAULT_TABLE table=test-table [id=a8039fee30694d148ff9c1162bfefae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:35:49.161382 16296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b6afd81fe949f6a9e77b0951a4ee34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:49.165462 16029 tablet_service.cc:1467] Processing CreateTablet for tablet 21b6afd81fe949f6a9e77b0951a4ee34 (DEFAULT_TABLE table=test-table [id=a8039fee30694d148ff9c1162bfefae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:35:49.166051 16163 tablet_service.cc:1467] Processing CreateTablet for tablet 21b6afd81fe949f6a9e77b0951a4ee34 (DEFAULT_TABLE table=test-table [id=a8039fee30694d148ff9c1162bfefae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:35:49.166707 16029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b6afd81fe949f6a9e77b0951a4ee34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:49.167388 16163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b6afd81fe949f6a9e77b0951a4ee34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:35:49.176832 16379 tablet_bootstrap.cc:492] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14: Bootstrap starting.
I20260925 10:35:49.179035 16380 tablet_bootstrap.cc:492] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb: Bootstrap starting.
I20260925 10:35:49.180560 16379 tablet_bootstrap.cc:654] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:49.181349 16381 tablet_bootstrap.cc:492] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b: Bootstrap starting.
I20260925 10:35:49.182019 16379 log.cc:824] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:49.182623 16380 tablet_bootstrap.cc:654] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:49.183815 16380 log.cc:824] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:49.184504 16381 tablet_bootstrap.cc:654] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b: Neither blocks nor log segments found. Creating new log.
I20260925 10:35:49.185715 16381 log.cc:824] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:35:49.185802 16379 tablet_bootstrap.cc:492] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14: No bootstrap required, opened a new log
I20260925 10:35:49.186077 16379 ts_tablet_manager.cc:1397] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260925 10:35:49.192873 16380 tablet_bootstrap.cc:492] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb: No bootstrap required, opened a new log
I20260925 10:35:49.192873 16381 tablet_bootstrap.cc:492] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b: No bootstrap required, opened a new log
I20260925 10:35:49.193143 16381 ts_tablet_manager.cc:1397] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260925 10:35:49.193143 16380 ts_tablet_manager.cc:1397] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260925 10:35:49.195945 16379 raft_consensus.cc:348] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:35:49.196368 16379 raft_consensus.cc:374] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:49.196507 16379 raft_consensus.cc:729] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5dc926b46da4f3db8c4f1e66852ad14, State: Initialized, Role: FOLLOWER
I20260925 10:35:49.197127 16379 consensus_queue.cc:260] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:35:49.198447 16361 heartbeater.cc:503] Master 127.0.97.62:46815 was elected leader, sending a full tablet report...
I20260925 10:35:49.200232 16379 ts_tablet_manager.cc:1428] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260925 10:35:49.203786 16381 raft_consensus.cc:348] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:35:49.203789 16380 raft_consensus.cc:348] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:35:49.204205 16381 raft_consensus.cc:374] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:49.204210 16380 raft_consensus.cc:374] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:35:49.204308 16381 raft_consensus.cc:729] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b469cc748964e6197a3734599e37e1b, State: Initialized, Role: FOLLOWER
I20260925 10:35:49.204308 16380 raft_consensus.cc:729] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a782401dd049179383645f3588efcb, State: Initialized, Role: FOLLOWER
I20260925 10:35:49.204905 16380 consensus_queue.cc:260] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:35:49.204905 16381 consensus_queue.cc:260] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:35:49.206377 16228 heartbeater.cc:503] Master 127.0.97.62:46815 was elected leader, sending a full tablet report...
I20260925 10:35:49.206733 16380 ts_tablet_manager.cc:1428] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260925 10:35:49.207011 16381 ts_tablet_manager.cc:1428] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b: Time spent starting tablet: real 0.014s user 0.014s sys 0.002s
W20260925 10:35:49.325172 16362 tablet.cc:2369] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:35:49.338676 16095 tablet.cc:2369] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:35:49.343536 16229 tablet.cc:2369] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:35:49.381790 16386 raft_consensus.cc:482] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:35:49.382097 16386 raft_consensus.cc:504] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:35:49.383756 16386 leader_election.cc:288] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5dc926b46da4f3db8c4f1e66852ad14 (127.0.97.3:34189), 76a782401dd049179383645f3588efcb (127.0.97.1:41979)
I20260925 10:35:49.390666 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.384155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54193 (local address 127.0.97.3:34189)
0925 10:35:49.384379 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:49.384385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:49.385062 (+ 677us) server_negotiation.cc:407] Connection header received
0925 10:35:49.385334 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:49.385339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:49.385417 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:49.385522 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:49.386221 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.387030 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.388881 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.389643 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.389736 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.389854 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:49.390234 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 10:35:49.390396 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:35:49.390674 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.384272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57927 (local address 127.0.97.1:41979)
0925 10:35:49.384501 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:49.384510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:49.385164 (+ 654us) server_negotiation.cc:407] Connection header received
0925 10:35:49.385331 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:49.385335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:49.385399 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:49.385465 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:35:49.386127 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.387120 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.388845 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.389674 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.389848 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.389945 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:49.390268 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 10:35:49.390420 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:35:49.390817 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.383956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34189 (local address 127.0.97.2:54193)
0925 10:35:49.384951 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:49.384966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:49.385144 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:49.385552 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:49.385557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:49.385578 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:49.386069 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:49.386082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.387159 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.387167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:49.388705 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:49.388716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.389871 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.389885 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:49.390052 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.390101 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 10:35:49.390173 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 10:35:49.390361 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":86,"thread_start_us":651,"threads_started":1}
I20260925 10:35:49.390846 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.384128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41979 (local address 127.0.97.2:57927)
0925 10:35:49.385065 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0925 10:35:49.385078 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:35:49.385179 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:35:49.385521 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:35:49.385525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:35:49.385546 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:35:49.385948 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:49.385957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.387256 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.387263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:49.388697 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:35:49.388709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.389942 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.389947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:35:49.390097 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.390119 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 10:35:49.390224 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 10:35:49.390412 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":83,"thread_start_us":609,"threads_started":1}
I20260925 10:35:49.392051 16316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b6afd81fe949f6a9e77b0951a4ee34" candidate_uuid: "6b469cc748964e6197a3734599e37e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" is_pre_election: true
I20260925 10:35:49.392051 16049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b6afd81fe949f6a9e77b0951a4ee34" candidate_uuid: "6b469cc748964e6197a3734599e37e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a782401dd049179383645f3588efcb" is_pre_election: true
I20260925 10:35:49.392611 16316 raft_consensus.cc:2454] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b469cc748964e6197a3734599e37e1b in term 0.
I20260925 10:35:49.392611 16049 raft_consensus.cc:2454] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b469cc748964e6197a3734599e37e1b in term 0.
I20260925 10:35:49.393287 16117 leader_election.cc:302] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b469cc748964e6197a3734599e37e1b, c5dc926b46da4f3db8c4f1e66852ad14; no voters:
I20260925 10:35:49.393685 16386 raft_consensus.cc:2781] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:35:49.393827 16386 raft_consensus.cc:482] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:35:49.393937 16386 raft_consensus.cc:3037] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:49.396051 16386 raft_consensus.cc:504] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:35:49.396631 16386 leader_election.cc:288] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [CANDIDATE]: Term 1 election: Requested vote from peers c5dc926b46da4f3db8c4f1e66852ad14 (127.0.97.3:34189), 76a782401dd049179383645f3588efcb (127.0.97.1:41979)
I20260925 10:35:49.396940 16316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b6afd81fe949f6a9e77b0951a4ee34" candidate_uuid: "6b469cc748964e6197a3734599e37e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5dc926b46da4f3db8c4f1e66852ad14"
I20260925 10:35:49.397149 16049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b6afd81fe949f6a9e77b0951a4ee34" candidate_uuid: "6b469cc748964e6197a3734599e37e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a782401dd049179383645f3588efcb"
I20260925 10:35:49.397176 16316 raft_consensus.cc:3037] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:49.397339 16049 raft_consensus.cc:3037] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:35:49.400357 16316 raft_consensus.cc:2454] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b469cc748964e6197a3734599e37e1b in term 1.
I20260925 10:35:49.400427 16049 raft_consensus.cc:2454] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b469cc748964e6197a3734599e37e1b in term 1.
I20260925 10:35:49.400772 16117 leader_election.cc:302] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b469cc748964e6197a3734599e37e1b, c5dc926b46da4f3db8c4f1e66852ad14; no voters:
I20260925 10:35:49.401046 16386 raft_consensus.cc:2781] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:35:49.401611 16386 raft_consensus.cc:686] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [term 1 LEADER]: Becoming Leader. State: Replica: 6b469cc748964e6197a3734599e37e1b, State: Running, Role: LEADER
I20260925 10:35:49.402148 16386 consensus_queue.cc:237] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:35:49.407231 15904 catalog_manager.cc:5495] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b469cc748964e6197a3734599e37e1b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6b469cc748964e6197a3734599e37e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } health_report { overall_health: UNKNOWN } } }
I20260925 10:35:49.448891 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.443996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41038 (local address 127.0.97.2:45481)
0925 10:35:49.444176 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:49.444180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:49.444289 (+ 109us) server_negotiation.cc:407] Connection header received
0925 10:35:49.444473 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:49.444476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:49.444551 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:49.444684 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:35:49.445005 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.445686 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.447084 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.447475 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.447568 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.447991 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:35:49.448455 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:35:49.448554 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:49.448642 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 10:35:49.448701 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:35:49.456691 16049 raft_consensus.cc:1261] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb [term 1 FOLLOWER]: Refusing update from remote peer 6b469cc748964e6197a3734599e37e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:35:49.456691 16316 raft_consensus.cc:1261] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14 [term 1 FOLLOWER]: Refusing update from remote peer 6b469cc748964e6197a3734599e37e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:35:49.457695 16390 consensus_queue.cc:1035] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:35:49.458001 16386 consensus_queue.cc:1035] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:35:49.483404 15904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:35:49.483739 15904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:35:49.487458 15904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Sending DeleteTablet for 3 replicas of tablet 21b6afd81fe949f6a9e77b0951a4ee34
I20260925 10:35:49.488533 16296 tablet_service.cc:1514] Processing DeleteTablet for tablet 21b6afd81fe949f6a9e77b0951a4ee34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:35:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47650
I20260925 10:35:49.488646 16029 tablet_service.cc:1514] Processing DeleteTablet for tablet 21b6afd81fe949f6a9e77b0951a4ee34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:35:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37034
I20260925 10:35:49.489352 16402 tablet_replica.cc:330] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14: stopping tablet replica
I20260925 10:35:49.489694 16402 raft_consensus.cc:2229] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:35:49.489729 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 15966
I20260925 10:35:49.490113 16402 raft_consensus.cc:2258] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:35:49.490863 16403 tablet_replica.cc:330] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb: stopping tablet replica
I20260925 10:35:49.491211 16403 raft_consensus.cc:2229] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:35:49.491604 16403 raft_consensus.cc:2258] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:35:49.491889 16163 tablet_service.cc:1514] Processing DeleteTablet for tablet 21b6afd81fe949f6a9e77b0951a4ee34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:35:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41022
I20260925 10:35:49.493839 16403 ts_tablet_manager.cc:1905] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:35:49.497273 16402 ts_tablet_manager.cc:1905] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:35:49.501979 16403 ts_tablet_manager.cc:1918] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:35:49.502106 16403 log.cc:1196] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/21b6afd81fe949f6a9e77b0951a4ee34
I20260925 10:35:49.502689 16403 ts_tablet_manager.cc:1939] T 21b6afd81fe949f6a9e77b0951a4ee34 P 76a782401dd049179383645f3588efcb: Deleting consensus metadata
I20260925 10:35:49.504737 15889 catalog_manager.cc:4841] TS 76a782401dd049179383645f3588efcb (127.0.97.1:41979): tablet 21b6afd81fe949f6a9e77b0951a4ee34 (table test-table [id=a8039fee30694d148ff9c1162bfefae7]) successfully deleted
I20260925 10:35:49.515043 16405 tablet_replica.cc:330] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b: stopping tablet replica
I20260925 10:35:49.515431 16405 raft_consensus.cc:2229] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:35:49.515877 16402 ts_tablet_manager.cc:1918] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:35:49.516039 16402 log.cc:1196] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/21b6afd81fe949f6a9e77b0951a4ee34
I20260925 10:35:49.516207 16405 raft_consensus.cc:2258] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:35:49.516551 16402 ts_tablet_manager.cc:1939] T 21b6afd81fe949f6a9e77b0951a4ee34 P c5dc926b46da4f3db8c4f1e66852ad14: Deleting consensus metadata
I20260925 10:35:49.518293 16405 ts_tablet_manager.cc:1905] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:35:49.518472 15890 catalog_manager.cc:4841] TS c5dc926b46da4f3db8c4f1e66852ad14 (127.0.97.3:34189): tablet 21b6afd81fe949f6a9e77b0951a4ee34 (table test-table [id=a8039fee30694d148ff9c1162bfefae7]) successfully deleted
I20260925 10:35:49.524602 16405 ts_tablet_manager.cc:1918] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:35:49.524776 16405 log.cc:1196] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/21b6afd81fe949f6a9e77b0951a4ee34
I20260925 10:35:49.525336 16405 ts_tablet_manager.cc:1939] T 21b6afd81fe949f6a9e77b0951a4ee34 P 6b469cc748964e6197a3734599e37e1b: Deleting consensus metadata
Sep 25 10:35:49 dist-test-slave-dxdd krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:35:49.526933 15892 catalog_manager.cc:4841] TS 6b469cc748964e6197a3734599e37e1b (127.0.97.2:45481): tablet 21b6afd81fe949f6a9e77b0951a4ee34 (table test-table [id=a8039fee30694d148ff9c1162bfefae7]) successfully deleted
I20260925 10:35:49.533655 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.491101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.0.97.1:41979)
0925 10:35:49.491263 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:49.491271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:49.509078 (+ 17807us) server_negotiation.cc:407] Connection header received
0925 10:35:49.509541 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:49.509547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:49.509628 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:49.509705 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:49.512007 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.513101 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.517168 (+ 4067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.517765 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.517876 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.527321 (+ 9445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:49.527335 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:49.527349 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:49.527385 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:49.530675 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:49.531495 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:49.531502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:49.531505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:49.531626 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:49.532050 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:49.532054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:49.532056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:49.532417 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:49.532700 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:49.533117 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 10:35:49.533284 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:35:49.534420 16089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:49.653767 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 15966
I20260925 10:35:49.671049 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 16100
Sep 25 10:35:49 dist-test-slave-dxdd krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:35:49.686936 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.671828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41040 (local address 127.0.97.2:45481)
0925 10:35:49.671992 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:49.672000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:49.672021 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:35:49.672253 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:49.672260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:49.672335 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:49.672444 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:49.673721 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.674627 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.675912 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.676318 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.676453 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.680981 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:49.680993 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:49.681006 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:49.681051 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:49.684059 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:49.684816 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:49.684822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:49.684824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:49.684955 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:49.685317 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:49.685320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:49.685321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:49.685654 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:49.685895 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:49.686167 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 10:35:49.686550 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:35:49.687700 16223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:49.797492 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 16100
I20260925 10:35:49.816265 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 16233
Sep 25 10:35:49 dist-test-slave-dxdd krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:35:49.830875 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:35:49.817046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.0.97.3:34189)
0925 10:35:49.817242 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:35:49.817247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:35:49.817277 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:35:49.817419 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:35:49.817423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:35:49.817483 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:35:49.817558 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:35:49.819052 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.819743 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.820970 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:35:49.821301 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:35:49.821380 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:35:49.825845 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:35:49.825858 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:35:49.825873 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:35:49.825903 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:35:49.828402 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:49.828979 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:49.828985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:49.828987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:49.829114 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:35:49.829437 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:35:49.829440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:35:49.829441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:35:49.829811 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:35:49.830017 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 10:35:49.830248 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 10:35:49.830527 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 10:35:49.831512 16356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:49.918349 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 16233
I20260925 10:35:49.936666 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 15873
I20260925 10:35:49.937783 15935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:35:50.044795 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 15873
I20260925 10:36:10.065699 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46815
--webserver_interface=127.0.97.62
--webserver_port=45773
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:36:10.391741 16410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:10.392005 16410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:10.392093 16410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:10.399257 16410 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:36:10.399351 16410 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:36:10.399415 16410 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:36:10.399488 16410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:36:10.399580 16410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:10.399698 16410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:10.399778 16410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:36:10.399859 16410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:36:10.409484 16410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:10.410734 16410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:10.411979 16410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:10.433888 16416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:10.433826 16417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:10.433866 16419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:10.439064 16410 server_base.cc:1034] running on GCE node
I20260925 10:36:10.439908 16410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:10.441430 16410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:10.442663 16410 hybrid_clock.cc:648] HybridClock initialized: now 1790332570442605 us; error 35 us; skew 500 ppm
I20260925 10:36:10.443032 16410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:36:10 dist-test-slave-dxdd krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:36:10.453052 16410 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:36:10.454806 16410 webserver.cc:460] Webserver started at http://127.0.97.62:45773/ using document root <none> and password file <none>
I20260925 10:36:10.455586 16410 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:10.455690 16410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:10.461501 16410 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 10:36:10.464308 16426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:10.465623 16410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:36:10.465807 16410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "98adecbb715843819b43c16086209ed1"
format_stamp: "Formatted at 2026-09-25 10:35:47 on dist-test-slave-dxdd"
server_key: "a1f396eb17d4bf33e2f3cd4e28a869d6"
server_key_iv: "536d95c5091d0755fbcc2853eaa197cd"
server_key_version: "encryptionkey@0"
I20260925 10:36:10.466974 16410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:10.482406 16410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:10.486876 16410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:10.487172 16410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:10.505702 16410 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46815
I20260925 10:36:10.505736 16477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46815 every 8 connection(s)
I20260925 10:36:10.507309 16410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:36:10.509104 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 16410
I20260925 10:36:10.509938 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41979
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46815
--webserver_port=36669
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:36:10.514613 16478 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:36:10.517836 16478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Bootstrap starting.
I20260925 10:36:10.537519 16478 log.cc:824] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:36:10.548818 16478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:36:10.549345 16478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Bootstrap complete.
I20260925 10:36:10.558112 16478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } }
I20260925 10:36:10.559034 16478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98adecbb715843819b43c16086209ed1, State: Initialized, Role: FOLLOWER
I20260925 10:36:10.559613 16478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } }
I20260925 10:36:10.559873 16478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:36:10.560022 16478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:36:10.560204 16478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:36:10.563915 16478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } }
I20260925 10:36:10.564411 16478 leader_election.cc:302] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98adecbb715843819b43c16086209ed1; no voters:
I20260925 10:36:10.565043 16478 leader_election.cc:288] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:36:10.565171 16482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:36:10.566444 16482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 2 LEADER]: Becoming Leader. State: Replica: 98adecbb715843819b43c16086209ed1, State: Running, Role: LEADER
I20260925 10:36:10.566956 16482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } }
I20260925 10:36:10.567267 16478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:36:10.569389 16483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "98adecbb715843819b43c16086209ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } } }
I20260925 10:36:10.569818 16483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:36:10.571277 16484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98adecbb715843819b43c16086209ed1. Latest consensus state: current_term: 2 leader_uuid: "98adecbb715843819b43c16086209ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } } }
I20260925 10:36:10.571734 16484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:36:10.572950 16488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:36:10.580757 16488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:36:10.582120 16488 catalog_manager.cc:1259] Loaded cluster ID: bf100fd3ddad4a0f940805b4068fbbc7
I20260925 10:36:10.582263 16488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:36:10.586731 16488 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:36:10.588289 16488 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Loaded TSK: 0
I20260925 10:36:10.640445 16488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Generated new TSK 1
I20260925 10:36:10.640842 16488 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:36:10.861966 16480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:10.862289 16480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:10.862504 16480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:10.870806 16480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:10.870977 16480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:10.871203 16480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:36:10.881973 16480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36669
--tserver_master_addrs=127.0.97.62:46815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:10.883306 16480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:10.884644 16480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:10.897827 16505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:10.897837 16506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:10.898540 16508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:10.898913 16480 server_base.cc:1034] running on GCE node
I20260925 10:36:10.899557 16480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:10.900497 16480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:10.901727 16480 hybrid_clock.cc:648] HybridClock initialized: now 1790332570901668 us; error 50 us; skew 500 ppm
I20260925 10:36:10.902078 16480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:36:10 dist-test-slave-dxdd krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:36:10.905220 16480 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:36:10.906899 16480 webserver.cc:460] Webserver started at http://127.0.97.1:36669/ using document root <none> and password file <none>
I20260925 10:36:10.907562 16480 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:10.907711 16480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:10.913323 16480 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:36:10.916137 16515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:10.917569 16480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:36:10.917723 16480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "76a782401dd049179383645f3588efcb"
format_stamp: "Formatted at 2026-09-25 10:35:48 on dist-test-slave-dxdd"
server_key: "da5f38e002ef3b13ee160f2372643af7"
server_key_iv: "f0e73c0cb771092015ac8869e003e07b"
server_key_version: "encryptionkey@0"
I20260925 10:36:10.918732 16480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:10.934664 16480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:10.939024 16480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:10.939311 16480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:10.940400 16480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:36:10.941818 16480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:36:10.941937 16480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:10.942058 16480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:36:10.942159 16480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:10.976797 16480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41979
I20260925 10:36:10.976876 16627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41979 every 8 connection(s)
I20260925 10:36:10.978221 16480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:36:10.982664 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 16480
I20260925 10:36:10.983242 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45481
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46815
--webserver_port=34343
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:36:10 dist-test-slave-dxdd krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:36:11.002002 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:10.981304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:52901)
0925 10:36:10.982011 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:10.982070 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:10.983968 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:10.985320 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:10.985331 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:10.985943 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:36:10.986571 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:10.986591 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:10.988289 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:10.988293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:10.989435 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:10.989446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:10.990596 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:10.990601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:10.990789 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:10.991481 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:36:10.991503 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:36:10.993586 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:36:10.998219 (+ 4633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:36:10.998225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:36:10.998235 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:36:10.998698 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:36:10.999169 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:36:10.999172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:36:10.999174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:36:10.999338 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:36:11.000254 (+ 916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:36:11.000259 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:36:11.000522 (+ 263us) client_negotiation.cc:769] Sending connection context
0925 10:36:11.000725 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 10:36:11.001080 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":289,"threads_started":1}
I20260925 10:36:11.002996 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:10.981001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52901 (local address 127.0.97.62:46815)
0925 10:36:10.981529 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:10.981540 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:10.982202 (+ 662us) server_negotiation.cc:407] Connection header received
0925 10:36:10.984509 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:10.984522 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:10.984831 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:10.985186 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:10.986767 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:10.988125 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:10.989576 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:10.990399 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:10.990583 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:10.993861 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:10.993875 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:10.993895 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:10.993948 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:10.998070 (+ 4122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:10.998838 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:10.998849 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:10.998860 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:10.999061 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:10.999459 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:10.999467 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:10.999479 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:11.000120 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:11.000332 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:11.001014 (+ 682us) server_negotiation.cc:299] Negotiation successful
0925 10:36:11.001484 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":223,"threads_started":1}
I20260925 10:36:11.005780 16628 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46815
I20260925 10:36:11.006091 16628 heartbeater.cc:464] Registering TS with master...
I20260925 10:36:11.006863 16628 heartbeater.cc:511] Master 127.0.97.62:46815 requested a full tablet report, sending...
I20260925 10:36:11.008912 16443 ts_manager.cc:194] Registered new tserver with Master: 76a782401dd049179383645f3588efcb (127.0.97.1:41979)
I20260925 10:36:11.010627 16443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52901
W20260925 10:36:11.303450 16632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:11.303750 16632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:11.303936 16632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:11.311010 16632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:11.311129 16632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:11.311324 16632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:36:11.320742 16632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34343
--tserver_master_addrs=127.0.97.62:46815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:11.321965 16632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:11.323261 16632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:11.335855 16638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:11.335928 16637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:11.335903 16640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:11.336794 16632 server_base.cc:1034] running on GCE node
I20260925 10:36:11.337319 16632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:11.338150 16632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:11.339341 16632 hybrid_clock.cc:648] HybridClock initialized: now 1790332571339309 us; error 61 us; skew 500 ppm
I20260925 10:36:11.339604 16632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:36:11 dist-test-slave-dxdd krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:36:11.342523 16632 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:36:11.343957 16632 webserver.cc:460] Webserver started at http://127.0.97.2:34343/ using document root <none> and password file <none>
I20260925 10:36:11.344615 16632 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:11.344774 16632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:11.350030 16632 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:36:11.352626 16647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:11.353850 16632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:36:11.354022 16632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "6b469cc748964e6197a3734599e37e1b"
format_stamp: "Formatted at 2026-09-25 10:35:48 on dist-test-slave-dxdd"
server_key: "930e9d6734836d25eca214d583bdd6f7"
server_key_iv: "f09828dc763dec5eef95751a1488fd80"
server_key_version: "encryptionkey@0"
I20260925 10:36:11.355031 16632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:11.374861 16632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:11.378756 16632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:11.379034 16632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:11.380018 16632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:36:11.381311 16632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:36:11.381433 16632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:11.381569 16632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:36:11.381656 16632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:11.413981 16632 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45481
I20260925 10:36:11.414062 16759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45481 every 8 connection(s)
I20260925 10:36:11.415380 16632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:36:11.416615 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 16632
I20260925 10:36:11.417353 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34189
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46815
--webserver_port=45451
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:36:11 dist-test-slave-dxdd krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:36:11.438035 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:11.418135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48185 (local address 127.0.97.62:46815)
0925 10:36:11.418355 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:11.418361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:11.419648 (+ 1287us) server_negotiation.cc:407] Connection header received
0925 10:36:11.422404 (+ 2756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:11.422409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:11.422477 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:11.422580 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:11.424590 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.425405 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.426987 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.427789 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.427930 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:11.431450 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:11.431458 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:11.431466 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:11.431510 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:11.434890 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:11.435573 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:11.435582 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:11.435588 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:11.435736 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:11.436128 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:11.436136 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:11.436142 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:11.436527 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:11.436695 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:11.437296 (+ 601us) server_negotiation.cc:299] Negotiation successful
0925 10:36:11.437554 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:36:11.438656 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:11.418659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:48185)
0925 10:36:11.419469 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:11.419518 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:11.422100 (+ 2582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:11.422842 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:11.422856 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:11.423580 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:36:11.424419 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:11.424447 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.425563 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.425567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:11.426839 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:11.426851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.428025 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.428030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:11.428221 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:11.428876 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:36:11.428902 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:36:11.431221 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:36:11.435017 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:36:11.435023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:36:11.435034 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:36:11.435429 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:36:11.435841 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:36:11.435844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:36:11.435846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:36:11.436009 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:36:11.436648 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:36:11.436654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:36:11.436941 (+ 287us) client_negotiation.cc:769] Sending connection context
0925 10:36:11.437150 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 10:36:11.437500 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":299,"threads_started":1}
I20260925 10:36:11.440107 16760 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46815
I20260925 10:36:11.440439 16760 heartbeater.cc:464] Registering TS with master...
I20260925 10:36:11.441376 16760 heartbeater.cc:511] Master 127.0.97.62:46815 requested a full tablet report, sending...
I20260925 10:36:11.442955 16443 ts_manager.cc:194] Registered new tserver with Master: 6b469cc748964e6197a3734599e37e1b (127.0.97.2:45481)
I20260925 10:36:11.443761 16443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48185
W20260925 10:36:11.747961 16763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:11.748216 16763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:11.748402 16763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:11.755674 16763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:11.755795 16763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:11.755973 16763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:36:11.765530 16763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45451
--tserver_master_addrs=127.0.97.62:46815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:11.766726 16763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:11.768021 16763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:11.780728 16771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:11.780789 16769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:11.780818 16768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:11.781804 16763 server_base.cc:1034] running on GCE node
I20260925 10:36:11.782436 16763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:11.783267 16763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:11.784475 16763 hybrid_clock.cc:648] HybridClock initialized: now 1790332571784437 us; error 28 us; skew 500 ppm
I20260925 10:36:11.784731 16763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:36:11 dist-test-slave-dxdd krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:36:11.787691 16763 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:36:11.789105 16763 webserver.cc:460] Webserver started at http://127.0.97.3:45451/ using document root <none> and password file <none>
I20260925 10:36:11.789708 16763 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:11.789816 16763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:11.794862 16763 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:36:11.797312 16778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:11.798625 16763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:36:11.798750 16763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "c5dc926b46da4f3db8c4f1e66852ad14"
format_stamp: "Formatted at 2026-09-25 10:35:48 on dist-test-slave-dxdd"
server_key: "d85ada91ed0ae8cc86fb630e4f1c60da"
server_key_iv: "02fba15dbb21f7876350ed95d698f01f"
server_key_version: "encryptionkey@0"
I20260925 10:36:11.799629 16763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:11.812665 16763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:11.816429 16763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:11.816645 16763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:11.817487 16763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:36:11.818730 16763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:36:11.818814 16763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:11.818889 16763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:36:11.818943 16763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:11.850973 16763 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34189
I20260925 10:36:11.851056 16890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34189 every 8 connection(s)
I20260925 10:36:11.852413 16763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:36:11.852969 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 16763
Sep 25 10:36:11 dist-test-slave-dxdd krb5kdc[15860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:36:11.870054 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:11.854121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.0.97.62:46815)
0925 10:36:11.854368 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:11.854374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:11.854735 (+ 361us) server_negotiation.cc:407] Connection header received
0925 10:36:11.854889 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:11.854897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:11.854991 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:11.855102 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:11.857209 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.858293 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.860012 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.860658 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.860802 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:11.863684 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:11.863689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:11.863692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:11.863723 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:11.866878 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:11.867815 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:11.867824 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:11.867830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:11.867975 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:11.868407 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:11.868411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:11.868413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:11.868783 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:11.869012 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:11.869388 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 10:36:11.869591 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 10:36:11.874969 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:11.855187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50895 (local address 127.0.97.62:46815)
0925 10:36:11.855674 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:11.855680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:11.856663 (+ 983us) server_negotiation.cc:407] Connection header received
0925 10:36:11.858972 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:11.858978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:11.859055 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:11.859175 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:11.860865 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.861959 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.863534 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.864050 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.864139 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:11.868285 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:11.868291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:11.868295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:11.868333 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:11.871912 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:11.872732 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:11.872738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:11.872743 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:11.872881 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:11.873348 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:11.873351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:11.873353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:11.873617 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:11.873731 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:11.874530 (+ 799us) server_negotiation.cc:299] Negotiation successful
0925 10:36:11.874695 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":286,"threads_started":1}
I20260925 10:36:11.876044 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:11.855708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:50895)
0925 10:36:11.856509 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:11.856557 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:11.858710 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:11.859439 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:11.859455 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:11.859963 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:36:11.860681 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:11.860705 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.862124 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.862127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:11.863389 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:11.863400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.864271 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.864281 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:11.864539 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:11.865125 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:36:11.865154 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:36:11.868017 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:36:11.872067 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:36:11.872078 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:36:11.872100 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:36:11.872594 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:36:11.872994 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:36:11.873000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:36:11.873004 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:36:11.873233 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:36:11.873751 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:36:11.873758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:36:11.874116 (+ 358us) client_negotiation.cc:769] Sending connection context
0925 10:36:11.874455 (+ 339us) client_negotiation.cc:240] Negotiation successful
0925 10:36:11.874867 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":352,"threads_started":1}
I20260925 10:36:11.877458 16891 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46815
I20260925 10:36:11.877804 16891 heartbeater.cc:464] Registering TS with master...
I20260925 10:36:11.878643 16891 heartbeater.cc:511] Master 127.0.97.62:46815 requested a full tablet report, sending...
I20260925 10:36:11.880071 16443 ts_manager.cc:194] Registered new tserver with Master: c5dc926b46da4f3db8c4f1e66852ad14 (127.0.97.3:34189)
I20260925 10:36:11.880898 16443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50895
I20260925 10:36:11.885252 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:36:11.892057 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:11.886064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.0.97.62:46815)
0925 10:36:11.886274 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:11.886282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:11.886744 (+ 462us) server_negotiation.cc:407] Connection header received
0925 10:36:11.886949 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:11.886953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:11.887094 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:11.887238 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:36:11.887683 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.888562 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.890166 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.890639 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.890772 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:11.891248 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:36:11.891435 (+ 187us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 10:36:11.891805 (+ 370us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56730: token expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 10:36:11.892196 16894 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56730: token expired
W20260925 10:36:11.892694 16365 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:46815)
I20260925 10:36:11.904779 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:11.893166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.0.97.62:46815)
0925 10:36:11.893310 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:11.893314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:11.893328 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:36:11.893449 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:11.893453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:11.893503 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:11.893636 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:11.894908 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.895634 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.896835 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.897207 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.897291 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:11.899786 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:11.899790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:11.899794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:11.899820 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:11.902060 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:11.902759 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:11.902763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:11.902766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:11.902926 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:11.903351 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:11.903362 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:11.903364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:11.903651 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:11.903925 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:11.904250 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 10:36:11.904452 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:36:11.907923 16443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:36:11.915130 16443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:36:11.948550 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:11.926986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58340 (local address 127.0.97.1:41979)
0925 10:36:11.939858 (+ 12872us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:11.939870 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:11.940821 (+ 951us) server_negotiation.cc:407] Connection header received
0925 10:36:11.941083 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:11.941091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:11.941286 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:11.941438 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:36:11.942474 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.943844 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.946185 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.947500 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.947633 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:11.947825 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:11.948050 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 10:36:11.948212 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12707,"thread_start_us":12563,"threads_started":1}
I20260925 10:36:11.949641 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:11.926698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38784 (local address 127.0.97.3:34189)
0925 10:36:11.939390 (+ 12692us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:11.939402 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:11.940532 (+ 1130us) server_negotiation.cc:407] Connection header received
0925 10:36:11.940843 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:11.940852 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:11.941122 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:11.941281 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:36:11.942429 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.944006 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.946928 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.948305 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.948441 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:11.948647 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:11.949144 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 10:36:11.949320 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12480,"thread_start_us":12311,"threads_started":1}
I20260925 10:36:11.949631 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:11.926575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41979 (local address 127.0.0.1:58340)
0925 10:36:11.940705 (+ 14130us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:11.940722 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:11.940907 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:11.941680 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:11.941687 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:11.941714 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:36:11.942251 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:11.942260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.944001 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.944006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:11.946029 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:11.946041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.947685 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.947689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:11.947825 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:11.947857 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:36:11.948005 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 10:36:11.948189 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13999,"mutex_wait_us":159,"thread_start_us":11603,"threads_started":1}
I20260925 10:36:11.949679 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:11.926037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34189 (local address 127.0.0.1:38784)
0925 10:36:11.940392 (+ 14355us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:11.940418 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:11.940636 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:11.941622 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:11.941629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:11.941662 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:36:11.942240 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:11.942252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.944142 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.944149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:11.946770 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:11.946784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.948795 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.948802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:11.948958 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:11.948990 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:36:11.949076 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 10:36:11.949210 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14199,"mutex_wait_us":284,"thread_start_us":14410,"threads_started":1}
I20260925 10:36:11.950314 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:11.925798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45481 (local address 127.0.0.1:60470)
0925 10:36:11.937712 (+ 11914us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:11.937770 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:11.937971 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:11.941563 (+ 3592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:11.941570 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:11.941598 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:36:11.942007 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:11.942017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.943957 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.943964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:11.946770 (+ 2806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:11.946784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.949591 (+ 2807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.949599 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:11.949784 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:11.949817 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 10:36:11.949896 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:36:11.950048 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11590,"mutex_wait_us":123,"thread_start_us":14303,"threads_started":1}
I20260925 10:36:11.951347 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:11.926025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60470 (local address 127.0.97.2:45481)
0925 10:36:11.941069 (+ 15044us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:11.941075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:11.941102 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:36:11.941181 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:11.941186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:11.941343 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:11.941551 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:36:11.942193 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.943814 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.947937 (+ 4123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:11.949345 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:11.950687 (+ 1342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:11.950862 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:11.950933 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 10:36:11.951094 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14904,"thread_start_us":14821,"threads_started":1}
I20260925 10:36:11.953722 16826 tablet_service.cc:1467] Processing CreateTablet for tablet f75f75a134d1488781228dae2337d6fc (DEFAULT_TABLE table=test-table [id=4dcabd94a3404ef9bb3f23751480866b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:36:11.953722 16563 tablet_service.cc:1467] Processing CreateTablet for tablet f75f75a134d1488781228dae2337d6fc (DEFAULT_TABLE table=test-table [id=4dcabd94a3404ef9bb3f23751480866b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:36:11.953969 16695 tablet_service.cc:1467] Processing CreateTablet for tablet f75f75a134d1488781228dae2337d6fc (DEFAULT_TABLE table=test-table [id=4dcabd94a3404ef9bb3f23751480866b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:36:11.955106 16695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75f75a134d1488781228dae2337d6fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:36:11.955121 16826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75f75a134d1488781228dae2337d6fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:36:11.955133 16563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75f75a134d1488781228dae2337d6fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:36:11.967403 16907 tablet_bootstrap.cc:492] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b: Bootstrap starting.
I20260925 10:36:11.969928 16908 tablet_bootstrap.cc:492] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb: Bootstrap starting.
I20260925 10:36:11.970808 16907 tablet_bootstrap.cc:654] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b: Neither blocks nor log segments found. Creating new log.
I20260925 10:36:11.972139 16907 log.cc:824] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:36:11.973677 16908 tablet_bootstrap.cc:654] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb: Neither blocks nor log segments found. Creating new log.
I20260925 10:36:11.974475 16909 tablet_bootstrap.cc:492] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14: Bootstrap starting.
I20260925 10:36:11.974895 16908 log.cc:824] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb: Log is configured to *not* fsync() on all Append() calls
I20260925 10:36:11.975517 16907 tablet_bootstrap.cc:492] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b: No bootstrap required, opened a new log
I20260925 10:36:11.975870 16907 ts_tablet_manager.cc:1397] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 10:36:11.977787 16909 tablet_bootstrap.cc:654] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14: Neither blocks nor log segments found. Creating new log.
I20260925 10:36:11.978964 16909 log.cc:824] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14: Log is configured to *not* fsync() on all Append() calls
I20260925 10:36:11.990393 16907 raft_consensus.cc:348] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:36:11.990980 16907 raft_consensus.cc:374] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:36:11.991114 16907 raft_consensus.cc:729] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b469cc748964e6197a3734599e37e1b, State: Initialized, Role: FOLLOWER
I20260925 10:36:11.992400 16907 consensus_queue.cc:260] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:36:11.994289 16908 tablet_bootstrap.cc:492] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb: No bootstrap required, opened a new log
I20260925 10:36:11.994277 16909 tablet_bootstrap.cc:492] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14: No bootstrap required, opened a new log
I20260925 10:36:11.994598 16909 ts_tablet_manager.cc:1397] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.006s
I20260925 10:36:11.994606 16908 ts_tablet_manager.cc:1397] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.011s
I20260925 10:36:12.003168 16760 heartbeater.cc:503] Master 127.0.97.62:46815 was elected leader, sending a full tablet report...
I20260925 10:36:12.003938 16907 ts_tablet_manager.cc:1428] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b: Time spent starting tablet: real 0.028s user 0.023s sys 0.004s
I20260925 10:36:12.005398 16909 raft_consensus.cc:348] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:36:12.005398 16908 raft_consensus.cc:348] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:36:12.005839 16908 raft_consensus.cc:374] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:36:12.005828 16909 raft_consensus.cc:374] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:36:12.005934 16909 raft_consensus.cc:729] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5dc926b46da4f3db8c4f1e66852ad14, State: Initialized, Role: FOLLOWER
I20260925 10:36:12.005932 16908 raft_consensus.cc:729] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a782401dd049179383645f3588efcb, State: Initialized, Role: FOLLOWER
I20260925 10:36:12.006628 16909 consensus_queue.cc:260] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:36:12.006628 16908 consensus_queue.cc:260] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:36:12.008147 16628 heartbeater.cc:503] Master 127.0.97.62:46815 was elected leader, sending a full tablet report...
I20260925 10:36:12.008204 16891 heartbeater.cc:503] Master 127.0.97.62:46815 was elected leader, sending a full tablet report...
I20260925 10:36:12.008723 16908 ts_tablet_manager.cc:1428] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 10:36:12.009486 16909 ts_tablet_manager.cc:1428] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
W20260925 10:36:12.104202 16892 tablet.cc:2369] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:36:12.148329 16915 raft_consensus.cc:482] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:36:12.148624 16915 raft_consensus.cc:504] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:36:12.150152 16915 leader_election.cc:288] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b469cc748964e6197a3734599e37e1b (127.0.97.2:45481), 76a782401dd049179383645f3588efcb (127.0.97.1:41979)
I20260925 10:36:12.155725 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:12.150686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41979 (local address 127.0.97.3:53133)
0925 10:36:12.150920 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:12.150936 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:12.151101 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:12.151436 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:12.151440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:12.151465 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:36:12.151668 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:12.151675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:12.152627 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:12.152634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:12.153891 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:12.153902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:12.154996 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:12.155001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:12.155190 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:12.155213 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:36:12.155317 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:36:12.155441 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 10:36:12.155721 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:12.150616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53133 (local address 127.0.97.1:41979)
0925 10:36:12.150817 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:12.150825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:12.151023 (+ 198us) server_negotiation.cc:407] Connection header received
0925 10:36:12.151250 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:12.151254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:12.151310 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:12.151416 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:36:12.151796 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:12.152503 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:12.154038 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:12.154843 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:12.154985 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:12.155075 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:12.155342 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 10:36:12.155473 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:36:12.156526 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:12.150673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52743 (local address 127.0.97.2:45481)
0925 10:36:12.150875 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:12.150883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:12.151709 (+ 826us) server_negotiation.cc:407] Connection header received
0925 10:36:12.151904 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:12.151908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:12.151963 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:12.152057 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:36:12.152561 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:12.153219 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:12.154906 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:12.155652 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:12.155808 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:12.155910 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:12.156208 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 10:36:12.156338 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:36:12.156625 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:12.150686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45481 (local address 127.0.97.3:52743)
0925 10:36:12.151612 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:12.151628 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:12.151761 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:12.152076 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:12.152080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:12.152103 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:36:12.152425 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:12.152433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:12.153356 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:12.153361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:12.154756 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:12.154767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:12.155859 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:12.155865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:12.156043 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:12.156073 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 10:36:12.156162 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:36:12.156250 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":218,"thread_start_us":409,"threads_started":1}
I20260925 10:36:12.156917 16583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75f75a134d1488781228dae2337d6fc" candidate_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a782401dd049179383645f3588efcb" is_pre_election: true
I20260925 10:36:12.157495 16583 raft_consensus.cc:2454] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5dc926b46da4f3db8c4f1e66852ad14 in term 0.
I20260925 10:36:12.157670 16715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75f75a134d1488781228dae2337d6fc" candidate_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b469cc748964e6197a3734599e37e1b" is_pre_election: true
I20260925 10:36:12.158149 16715 raft_consensus.cc:2454] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5dc926b46da4f3db8c4f1e66852ad14 in term 0.
I20260925 10:36:12.158200 16779 leader_election.cc:302] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76a782401dd049179383645f3588efcb, c5dc926b46da4f3db8c4f1e66852ad14; no voters:
I20260925 10:36:12.158655 16915 raft_consensus.cc:2781] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:36:12.158810 16915 raft_consensus.cc:482] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:36:12.158921 16915 raft_consensus.cc:3037] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:36:12.161022 16915 raft_consensus.cc:504] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
I20260925 10:36:12.161625 16915 leader_election.cc:288] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [CANDIDATE]: Term 1 election: Requested vote from peers 6b469cc748964e6197a3734599e37e1b (127.0.97.2:45481), 76a782401dd049179383645f3588efcb (127.0.97.1:41979)
I20260925 10:36:12.162019 16715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75f75a134d1488781228dae2337d6fc" candidate_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b469cc748964e6197a3734599e37e1b"
I20260925 10:36:12.162091 16583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75f75a134d1488781228dae2337d6fc" candidate_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a782401dd049179383645f3588efcb"
I20260925 10:36:12.162189 16715 raft_consensus.cc:3037] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:36:12.162350 16583 raft_consensus.cc:3037] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:36:12.164359 16715 raft_consensus.cc:2454] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5dc926b46da4f3db8c4f1e66852ad14 in term 1.
I20260925 10:36:12.164521 16583 raft_consensus.cc:2454] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5dc926b46da4f3db8c4f1e66852ad14 in term 1.
I20260925 10:36:12.164781 16782 leader_election.cc:302] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b469cc748964e6197a3734599e37e1b, c5dc926b46da4f3db8c4f1e66852ad14; no voters:
I20260925 10:36:12.165104 16915 raft_consensus.cc:2781] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:36:12.165472 16915 raft_consensus.cc:686] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [term 1 LEADER]: Becoming Leader. State: Replica: c5dc926b46da4f3db8c4f1e66852ad14, State: Running, Role: LEADER
I20260925 10:36:12.165899 16915 consensus_queue.cc:237] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } } peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } }
W20260925 10:36:12.167220 16761 tablet.cc:2369] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:36:12.170900 16443 catalog_manager.cc:5495] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5dc926b46da4f3db8c4f1e66852ad14 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5dc926b46da4f3db8c4f1e66852ad14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 } health_report { overall_health: UNKNOWN } } }
W20260925 10:36:12.230897 16629 tablet.cc:2369] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:36:12.237432 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:12.232251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.0.97.3:34189)
0925 10:36:12.232473 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:12.232480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:12.232499 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:36:12.232709 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:12.232712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:12.232783 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:12.232933 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:36:12.233295 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:12.234014 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:12.235407 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:12.235797 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:12.235926 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:12.236405 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:36:12.236865 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:36:12.237026 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:12.237081 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 10:36:12.237153 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 10:36:12.245424 16583 raft_consensus.cc:1261] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb [term 1 FOLLOWER]: Refusing update from remote peer c5dc926b46da4f3db8c4f1e66852ad14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:36:12.245482 16715 raft_consensus.cc:1261] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b [term 1 FOLLOWER]: Refusing update from remote peer c5dc926b46da4f3db8c4f1e66852ad14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:36:12.246392 16917 consensus_queue.cc:1035] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b469cc748964e6197a3734599e37e1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:36:12.246731 16915 consensus_queue.cc:1035] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a782401dd049179383645f3588efcb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:36:12.258503 16920 mvcc.cc:205] Tried to move back new op lower bound from 7333202215909318656 to 7333202215596163072. Current Snapshot: MvccSnapshot[applied={T|T < 7333202215909318656}]
I20260925 10:36:12.286840 16443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:36:12.287120 16443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:36:12.290758 16443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Sending DeleteTablet for 3 replicas of tablet f75f75a134d1488781228dae2337d6fc
I20260925 10:36:12.291862 16695 tablet_service.cc:1514] Processing DeleteTablet for tablet f75f75a134d1488781228dae2337d6fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:36:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60470
I20260925 10:36:12.291951 16563 tablet_service.cc:1514] Processing DeleteTablet for tablet f75f75a134d1488781228dae2337d6fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:36:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58340
I20260925 10:36:12.292120 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 16480
I20260925 10:36:12.292793 16930 tablet_replica.cc:330] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b: stopping tablet replica
I20260925 10:36:12.292889 16826 tablet_service.cc:1514] Processing DeleteTablet for tablet f75f75a134d1488781228dae2337d6fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:36:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38784
I20260925 10:36:12.293414 16931 tablet_replica.cc:330] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb: stopping tablet replica
I20260925 10:36:12.293871 16931 raft_consensus.cc:2229] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:36:12.294793 16930 raft_consensus.cc:2229] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:36:12.295370 16931 raft_consensus.cc:2258] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:36:12.295842 16930 raft_consensus.cc:2258] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:36:12.298574 16931 ts_tablet_manager.cc:1905] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:36:12.305202 16932 tablet_replica.cc:330] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14: stopping tablet replica
I20260925 10:36:12.305198 16930 ts_tablet_manager.cc:1905] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:36:12.305583 16932 raft_consensus.cc:2229] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:36:12.306342 16931 ts_tablet_manager.cc:1918] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:36:12.306293 16932 raft_consensus.cc:2258] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:36:12.306677 16931 log.cc:1196] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/f75f75a134d1488781228dae2337d6fc
I20260925 10:36:12.307495 16931 ts_tablet_manager.cc:1939] T f75f75a134d1488781228dae2337d6fc P 76a782401dd049179383645f3588efcb: Deleting consensus metadata
I20260925 10:36:12.308274 16932 ts_tablet_manager.cc:1905] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:36:12.309525 16427 catalog_manager.cc:4841] TS 76a782401dd049179383645f3588efcb (127.0.97.1:41979): tablet f75f75a134d1488781228dae2337d6fc (table test-table [id=4dcabd94a3404ef9bb3f23751480866b]) successfully deleted
I20260925 10:36:12.311326 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:12.293578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.0.97.1:41979)
0925 10:36:12.293724 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:12.293730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:12.293748 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:36:12.293804 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:12.293819 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:12.293924 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:12.294022 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:12.295944 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:12.297164 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:12.298995 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:12.299629 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:12.299769 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:12.303219 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:12.303231 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:12.303252 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:12.303300 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:12.307290 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:12.308176 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:12.308185 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:12.308188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:12.308338 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:12.308849 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:12.308854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:12.308857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:12.309329 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:12.309604 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:12.310751 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0925 10:36:12.310978 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:36:12.312170 16930 ts_tablet_manager.cc:1918] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:36:12.312251 16623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:36:12.312327 16930 log.cc:1196] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/f75f75a134d1488781228dae2337d6fc
I20260925 10:36:12.313038 16930 ts_tablet_manager.cc:1939] T f75f75a134d1488781228dae2337d6fc P 6b469cc748964e6197a3734599e37e1b: Deleting consensus metadata
I20260925 10:36:12.314828 16430 catalog_manager.cc:4841] TS 6b469cc748964e6197a3734599e37e1b (127.0.97.2:45481): tablet f75f75a134d1488781228dae2337d6fc (table test-table [id=4dcabd94a3404ef9bb3f23751480866b]) successfully deleted
I20260925 10:36:12.315606 16932 ts_tablet_manager.cc:1918] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:36:12.315793 16932 log.cc:1196] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/f75f75a134d1488781228dae2337d6fc
I20260925 10:36:12.316316 16932 ts_tablet_manager.cc:1939] T f75f75a134d1488781228dae2337d6fc P c5dc926b46da4f3db8c4f1e66852ad14: Deleting consensus metadata
I20260925 10:36:12.317653 16428 catalog_manager.cc:4841] TS c5dc926b46da4f3db8c4f1e66852ad14 (127.0.97.3:34189): tablet f75f75a134d1488781228dae2337d6fc (table test-table [id=4dcabd94a3404ef9bb3f23751480866b]) successfully deleted
I20260925 10:36:12.406303 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 16480
I20260925 10:36:12.429664 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 16632
I20260925 10:36:12.444410 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:12.430546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60474 (local address 127.0.97.2:45481)
0925 10:36:12.430755 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:12.430763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:12.430785 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:36:12.430847 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:12.430855 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:12.430946 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:12.431053 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:12.432281 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:12.433441 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:12.434898 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:12.435523 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:12.435652 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:12.438122 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:12.438135 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:12.438153 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:12.438196 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:12.441747 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:12.442443 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:12.442450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:12.442452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:12.442569 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:12.442935 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:12.442945 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:12.442947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:12.443302 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:12.443594 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:12.443882 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 10:36:12.444089 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:36:12.445065 16755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:36:12.537822 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 16632
I20260925 10:36:12.555171 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 16763
I20260925 10:36:12.571345 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:12.555960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38804 (local address 127.0.97.3:34189)
0925 10:36:12.556158 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:12.556164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:12.556188 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:36:12.556351 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:12.556357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:12.556440 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:12.556554 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:12.557816 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:12.559086 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:12.560531 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:12.561241 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:12.561381 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:12.564134 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:12.564146 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:12.564163 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:12.564212 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:12.568388 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:12.569069 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:12.569079 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:12.569083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:12.569263 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:12.569613 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:12.569618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:12.569621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:12.570095 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:12.570448 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:12.570684 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 10:36:12.570924 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 10:36:12.572146 16886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:36:12.691615 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 16763
I20260925 10:36:12.711068 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 16410
I20260925 10:36:12.712298 16473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:36:12.827708 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 16410
I20260925 10:36:32.849303 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46815
--webserver_interface=127.0.97.62
--webserver_port=45773
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:36:33.166966 16937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:33.167224 16937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:33.167429 16937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:33.174201 16937 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:36:33.174275 16937 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:36:33.174346 16937 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:36:33.174425 16937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:36:33.174512 16937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:33.174626 16937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:33.174690 16937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:36:33.174780 16937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:36:33.184172 16937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:33.185400 16937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:33.186753 16937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:33.198648 16942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:33.198693 16945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:33.198730 16943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:33.199788 16937 server_base.cc:1034] running on GCE node
I20260925 10:36:33.200505 16937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:33.201833 16937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:33.203068 16937 hybrid_clock.cc:648] HybridClock initialized: now 1790332593202997 us; error 48 us; skew 500 ppm
I20260925 10:36:33.203394 16937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:36:33 dist-test-slave-dxdd krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:36:33.210626 16937 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:36:33.212003 16937 webserver.cc:460] Webserver started at http://127.0.97.62:45773/ using document root <none> and password file <none>
I20260925 10:36:33.212670 16937 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:33.212795 16937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:33.218039 16937 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:36:33.220896 16952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:33.222147 16937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 10:36:33.222371 16937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "98adecbb715843819b43c16086209ed1"
format_stamp: "Formatted at 2026-09-25 10:35:47 on dist-test-slave-dxdd"
server_key: "a1f396eb17d4bf33e2f3cd4e28a869d6"
server_key_iv: "536d95c5091d0755fbcc2853eaa197cd"
server_key_version: "encryptionkey@0"
I20260925 10:36:33.223269 16937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:33.237161 16937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:33.241307 16937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:33.241575 16937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:33.259054 16937 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46815
I20260925 10:36:33.259090 17003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46815 every 8 connection(s)
I20260925 10:36:33.260438 16937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:36:33.261193 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 16937
I20260925 10:36:33.262143 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41979
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46815
--webserver_port=36669
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:36:33.267339 17004 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:36:33.270754 17004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Bootstrap starting.
I20260925 10:36:33.290515 17004 log.cc:824] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:36:33.305819 17004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:36:33.306349 17004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Bootstrap complete.
I20260925 10:36:33.316262 17004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } }
I20260925 10:36:33.317145 17004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98adecbb715843819b43c16086209ed1, State: Initialized, Role: FOLLOWER
I20260925 10:36:33.317821 17004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } }
I20260925 10:36:33.318061 17004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:36:33.318178 17004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:36:33.318352 17004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:36:33.321762 17004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } }
I20260925 10:36:33.322222 17004 leader_election.cc:302] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98adecbb715843819b43c16086209ed1; no voters:
I20260925 10:36:33.322909 17004 leader_election.cc:288] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 10:36:33.323038 17008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 10:36:33.324270 17008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [term 3 LEADER]: Becoming Leader. State: Replica: 98adecbb715843819b43c16086209ed1, State: Running, Role: LEADER
I20260925 10:36:33.324800 17008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } }
I20260925 10:36:33.325052 17004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:36:33.331933 17009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "98adecbb715843819b43c16086209ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } } }
I20260925 10:36:33.332299 17009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:36:33.332839 17010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98adecbb715843819b43c16086209ed1. Latest consensus state: current_term: 3 leader_uuid: "98adecbb715843819b43c16086209ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98adecbb715843819b43c16086209ed1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46815 } } }
I20260925 10:36:33.333119 17010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:36:33.346434 17018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:36:33.352615 17018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:36:33.353691 17018 catalog_manager.cc:1259] Loaded cluster ID: bf100fd3ddad4a0f940805b4068fbbc7
I20260925 10:36:33.353796 17018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:36:33.356498 17018 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:36:33.357611 17018 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Loaded TSK: 0
I20260925 10:36:33.357751 17018 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 98adecbb715843819b43c16086209ed1: Loaded TSK: 1
F20260925 10:36:33.358582 17018 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3d6a6579ad google::LogMessage::Fail() at ??:0
@ 0x7f3d6a65a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f3d6a6574de google::LogMessage::Flush() at ??:0
@ 0x7f3d6a658a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3d6e3bb7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f3d6e3aa73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f3d800861c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f3d80085783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f3d801112a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f3d801110c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f3d8015b5e9 std::function<>::operator()() at ??:0
@ 0x7f3d80089cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f3d80087c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3d8010f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f3d8010f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3d6d2a3345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3d6d2a691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f3d6d2a6799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3d6d281eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3d7055b6db start_thread at ??:0
@ 0x7f3d694f161f clone at ??:0
W20260925 10:36:33.632326 17006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:33.632637 17006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:33.632731 17006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:33.639719 17006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:33.639811 17006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:33.639923 17006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:36:33.649394 17006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36669
--tserver_master_addrs=127.0.97.62:46815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:33.650584 17006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:33.651846 17006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:33.664580 17052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:33.664760 17051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:33.664808 17054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:33.665728 17006 server_base.cc:1034] running on GCE node
I20260925 10:36:33.666244 17006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:33.667260 17006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:33.668452 17006 hybrid_clock.cc:648] HybridClock initialized: now 1790332593668416 us; error 27 us; skew 500 ppm
I20260925 10:36:33.668707 17006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:36:33 dist-test-slave-dxdd krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:36:33.671603 17006 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:36:33.672947 17006 webserver.cc:460] Webserver started at http://127.0.97.1:36669/ using document root <none> and password file <none>
I20260925 10:36:33.673525 17006 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:33.673621 17006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:33.678543 17006 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260925 10:36:33.681001 17061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:33.682186 17006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:36:33.682380 17006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "76a782401dd049179383645f3588efcb"
format_stamp: "Formatted at 2026-09-25 10:35:48 on dist-test-slave-dxdd"
server_key: "da5f38e002ef3b13ee160f2372643af7"
server_key_iv: "f0e73c0cb771092015ac8869e003e07b"
server_key_version: "encryptionkey@0"
I20260925 10:36:33.683266 17006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:33.709074 17006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:33.712755 17006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:33.712975 17006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:33.713860 17006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:36:33.715047 17006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:36:33.715139 17006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:33.715219 17006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:36:33.715281 17006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:33.747360 17006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41979
I20260925 10:36:33.747459 17173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41979 every 8 connection(s)
I20260925 10:36:33.748735 17006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:36:33.753472 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:33.751684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:54789)
0925 10:36:33.752448 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:33.752657 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":341,"threads_started":1}
W20260925 10:36:33.754175 17174 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46815 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46815: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
I20260925 10:36:33.754597 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 17006
I20260925 10:36:33.755043 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:33.754732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:50149)
0925 10:36:33.754871 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:33.754941 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 10:36:33.755272 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45481
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46815
--webserver_port=34343
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:36:33.756031 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:33.755606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:44333)
0925 10:36:33.755770 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:33.755870 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 10:36:33.756264 17174 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:36:34.070295 17177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:34.070557 17177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:34.070761 17177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:34.077638 17177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:34.077746 17177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:34.077852 17177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:36:34.087288 17177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34343
--tserver_master_addrs=127.0.97.62:46815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:34.088505 17177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:34.089776 17177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:34.102154 17183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:34.102201 17185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:34.102231 17182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:34.103068 17177 server_base.cc:1034] running on GCE node
I20260925 10:36:34.103554 17177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:34.104333 17177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:34.105520 17177 hybrid_clock.cc:648] HybridClock initialized: now 1790332594105478 us; error 35 us; skew 500 ppm
I20260925 10:36:34.105829 17177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:36:34 dist-test-slave-dxdd krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:36:34.108814 17177 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:36:34.110196 17177 webserver.cc:460] Webserver started at http://127.0.97.2:34343/ using document root <none> and password file <none>
I20260925 10:36:34.110841 17177 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:34.110935 17177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:34.115675 17177 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:36:34.118046 17192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:34.119275 17177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:36:34.119439 17177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "6b469cc748964e6197a3734599e37e1b"
format_stamp: "Formatted at 2026-09-25 10:35:48 on dist-test-slave-dxdd"
server_key: "930e9d6734836d25eca214d583bdd6f7"
server_key_iv: "f09828dc763dec5eef95751a1488fd80"
server_key_version: "encryptionkey@0"
I20260925 10:36:34.120325 17177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:34.150444 17177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:34.154098 17177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:34.154382 17177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:34.155292 17177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:36:34.156538 17177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:36:34.156665 17177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:34.156793 17177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:36:34.156886 17177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:34.189186 17177 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45481
I20260925 10:36:34.189270 17304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45481 every 8 connection(s)
I20260925 10:36:34.190668 17177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:36:34.195364 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:34.193485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:39723)
0925 10:36:34.194187 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:34.194460 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":277,"threads_started":1}
W20260925 10:36:34.195971 17305 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46815 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46815: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
I20260925 10:36:34.196686 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:34.196413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:53659)
0925 10:36:34.196536 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:34.196594 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 10:36:34.197443 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:34.197116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:38723)
0925 10:36:34.197230 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:34.197294 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 10:36:34.197691 17305 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:36:34.197845 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 17177
I20260925 10:36:34.198407 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34189
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46815
--webserver_port=45451
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:36:34.509181 17308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:34.509447 17308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:34.509656 17308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:34.516557 17308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:34.516655 17308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:34.516846 17308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:36:34.526453 17308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45451
--tserver_master_addrs=127.0.97.62:46815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:34.527658 17308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:34.528908 17308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:34.541039 17313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:34.541049 17314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:34.541254 17316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:34.542107 17308 server_base.cc:1034] running on GCE node
I20260925 10:36:34.542698 17308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:34.543507 17308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:34.544727 17308 hybrid_clock.cc:648] HybridClock initialized: now 1790332594544687 us; error 73 us; skew 500 ppm
I20260925 10:36:34.545015 17308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:36:34 dist-test-slave-dxdd krb5kdc[15860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:36:34.547941 17308 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:36:34.549321 17308 webserver.cc:460] Webserver started at http://127.0.97.3:45451/ using document root <none> and password file <none>
I20260925 10:36:34.549904 17308 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:34.549998 17308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:34.554797 17308 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:36:34.557171 17323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:34.558431 17308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260925 10:36:34.558553 17308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "c5dc926b46da4f3db8c4f1e66852ad14"
format_stamp: "Formatted at 2026-09-25 10:35:48 on dist-test-slave-dxdd"
server_key: "d85ada91ed0ae8cc86fb630e4f1c60da"
server_key_iv: "02fba15dbb21f7876350ed95d698f01f"
server_key_version: "encryptionkey@0"
I20260925 10:36:34.559392 17308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:34.581748 17308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:34.585937 17308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:34.586256 17308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:34.587402 17308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:36:34.588701 17308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:36:34.588819 17308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:34.588950 17308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:36:34.589061 17308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:34.621769 17308 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34189
I20260925 10:36:34.621833 17435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34189 every 8 connection(s)
I20260925 10:36:34.623415 17308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:36:34.628108 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:34.626280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:34101)
0925 10:36:34.627063 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:34.627262 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":279,"threads_started":1}
W20260925 10:36:34.628736 17436 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46815 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46815: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
I20260925 10:36:34.629431 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:34.629160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:49757)
0925 10:36:34.629280 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:34.629338 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 10:36:34.630206 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 17308
I20260925 10:36:34.630461 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:34.630140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:51545)
0925 10:36:34.630251 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:34.630359 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:36:34.630684 17436 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:36:34.757483 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:34.756762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:43833)
0925 10:36:34.757189 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:34.757268 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":213,"threads_started":1}
I20260925 10:36:35.198925 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:35.198151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:60219)
0925 10:36:35.198618 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:35.198714 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":266,"threads_started":1}
I20260925 10:36:35.632117 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:35.631220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:53639)
0925 10:36:35.631759 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:35.631842 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":288,"threads_started":1}
I20260925 10:36:35.759217 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:35.758531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:49595)
0925 10:36:35.758980 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:35.759056 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":238,"threads_started":1}
I20260925 10:36:36.200408 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:36.199752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:59993)
0925 10:36:36.200186 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:36.200256 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":230,"threads_started":1}
I20260925 10:36:36.633810 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:36.632958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:41543)
0925 10:36:36.633513 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:36.633588 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":309,"threads_started":1}
I20260925 10:36:36.760632 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:36.760000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:50557)
0925 10:36:36.760391 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:36.760463 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":208,"threads_started":1}
I20260925 10:36:37.202163 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:37.201470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:34221)
0925 10:36:37.201917 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:37.201997 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":244,"threads_started":1}
I20260925 10:36:37.635612 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:37.634826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:60501)
0925 10:36:37.635367 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:37.635436 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":288,"threads_started":1}
I20260925 10:36:37.762416 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:37.761575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:55837)
0925 10:36:37.762054 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:37.762177 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":235,"threads_started":1}
I20260925 10:36:38.203716 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:38.203029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:40647)
0925 10:36:38.203455 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:38.203529 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":225,"threads_started":1}
I20260925 10:36:38.637125 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:38.636432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:39855)
0925 10:36:38.636852 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:38.636925 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":225,"threads_started":1}
I20260925 10:36:38.763924 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:38.763226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:52589)
0925 10:36:38.763677 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:38.763768 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":244,"threads_started":1}
I20260925 10:36:39.205210 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:39.204551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:36255)
0925 10:36:39.204978 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:39.205052 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":233,"threads_started":1}
I20260925 10:36:39.638757 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:39.638026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:51193)
0925 10:36:39.638533 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:39.638604 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":230,"threads_started":1}
I20260925 10:36:39.765384 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:39.764675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:41255)
0925 10:36:39.765115 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:39.765231 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":231,"threads_started":1}
I20260925 10:36:40.206771 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:40.206026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:44399)
0925 10:36:40.206531 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:40.206614 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":294,"threads_started":1}
I20260925 10:36:40.640218 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:40.639536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:41653)
0925 10:36:40.639962 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:40.640068 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":235,"threads_started":1}
I20260925 10:36:40.766940 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:40.766161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:37599)
0925 10:36:40.766667 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:40.766740 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":249,"threads_started":1}
I20260925 10:36:41.208300 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:41.207584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:40629)
0925 10:36:41.208018 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:41.208098 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":188,"threads_started":1}
I20260925 10:36:41.641739 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:41.641031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:38597)
0925 10:36:41.641468 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:41.641538 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":197,"threads_started":1}
I20260925 10:36:41.768430 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:41.767778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:56247)
0925 10:36:41.768196 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:41.768268 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":219,"threads_started":1}
I20260925 10:36:42.209830 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:42.209203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:47247)
0925 10:36:42.209608 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:42.209677 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":203,"threads_started":1}
I20260925 10:36:42.643463 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:42.642715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:38483)
0925 10:36:42.643231 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:42.643311 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":285,"threads_started":1}
I20260925 10:36:42.770143 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:42.769407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:42371)
0925 10:36:42.769878 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:42.769963 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":228,"threads_started":1}
I20260925 10:36:43.211480 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:43.210760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:48627)
0925 10:36:43.211242 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:43.211320 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":232,"threads_started":1}
I20260925 10:36:43.644939 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:43.644201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:59715)
0925 10:36:43.644700 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:43.644773 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":275,"threads_started":1}
I20260925 10:36:43.771703 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:43.771028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:57413)
0925 10:36:43.771474 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:43.771549 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":255,"threads_started":1}
I20260925 10:36:44.213093 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:44.212462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:56567)
0925 10:36:44.212871 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:44.212942 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":224,"threads_started":1}
I20260925 10:36:44.646540 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:44.645814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:53675)
0925 10:36:44.646242 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:44.646381 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":230,"threads_started":1}
I20260925 10:36:44.773248 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:44.772569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:41233)
0925 10:36:44.772998 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:44.773081 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":234,"threads_started":1}
I20260925 10:36:45.214577 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:45.213900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:48397)
0925 10:36:45.214321 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:45.214404 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":204,"threads_started":1}
I20260925 10:36:45.647985 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:45.647348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:44127)
0925 10:36:45.647744 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:45.647822 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":201,"threads_started":1}
I20260925 10:36:45.775138 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:45.774284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:41439)
0925 10:36:45.774847 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:45.774951 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":273,"threads_started":1}
I20260925 10:36:46.215979 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:46.215378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:51801)
0925 10:36:46.215758 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:46.215829 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":198,"threads_started":1}
I20260925 10:36:46.649581 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:46.648792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:41369)
0925 10:36:46.649328 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:46.649437 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":290,"threads_started":1}
I20260925 10:36:46.776683 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:46.776016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:50523)
0925 10:36:46.776432 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:46.776509 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":206,"threads_started":1}
I20260925 10:36:47.217639 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:47.216884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:43779)
0925 10:36:47.217393 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:47.217469 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":292,"threads_started":1}
I20260925 10:36:47.651078 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:47.650429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:39713)
0925 10:36:47.650833 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:47.650906 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":195,"threads_started":1}
I20260925 10:36:47.778189 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:47.777525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:39941)
0925 10:36:47.777946 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:47.778020 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":218,"threads_started":1}
I20260925 10:36:48.219159 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:48.218501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:46221)
0925 10:36:48.218905 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:48.218976 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":199,"threads_started":1}
I20260925 10:36:48.652815 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:48.652068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:57329)
0925 10:36:48.652546 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:48.652616 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":270,"threads_started":1}
I20260925 10:36:48.779829 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:48.779178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.1:35581)
0925 10:36:48.779576 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:48.779655 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":206,"threads_started":1}
I20260925 10:36:49.220852 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:49.220152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.2:40671)
0925 10:36:49.220591 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:49.220675 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":235,"threads_started":1}
I20260925 10:36:49.637346 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 17006
I20260925 10:36:49.654877 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:49.654045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46815 (local address 127.0.97.3:57943)
0925 10:36:49.654542 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:49.654625 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":200,"threads_started":1}
I20260925 10:36:49.662431 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:49.638558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.0.97.1:41979)
0925 10:36:49.639108 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:49.639123 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:49.639182 (+ 59us) server_negotiation.cc:407] Connection header received
0925 10:36:49.640682 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:49.640691 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:49.641109 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:49.641463 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:49.642955 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:49.644188 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:49.645452 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:49.646108 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:49.646292 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:49.656562 (+ 10270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:49.656569 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:49.656584 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:49.656618 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:49.659291 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:49.659948 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:49.659954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:49.659960 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:49.660112 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:49.660483 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:49.660492 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:49.660494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:49.660971 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:49.661275 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:49.661718 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 10:36:49.662117 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":234,"threads_started":1}
I20260925 10:36:49.663909 17169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:36:49.744953 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 17006
I20260925 10:36:49.761519 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 17177
I20260925 10:36:49.778110 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:49.762557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.0.97.2:45481)
0925 10:36:49.763110 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:49.763124 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:49.763162 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:36:49.764835 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:49.764845 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:49.765262 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:49.765560 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:49.766995 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:49.768372 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:49.769499 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:49.770137 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:49.770268 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:49.772650 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:49.772657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:49.772670 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:49.772721 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:49.775390 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:49.776017 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:49.776023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:49.776028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:49.776173 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:49.776465 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:49.776468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:49.776470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:49.776920 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:49.777129 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:49.777453 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 10:36:49.777836 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":287,"threads_started":1}
I20260925 10:36:49.779419 17300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:36:49.860249 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 17177
I20260925 10:36:49.876817 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 17308
I20260925 10:36:49.894726 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:49.877742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35186 (local address 127.0.97.3:34189)
0925 10:36:49.878472 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:49.878487 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:49.878529 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:36:49.880085 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:49.880093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:49.880509 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:49.880844 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:49.882243 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:49.883533 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:49.884850 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:49.885532 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:49.885726 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:49.888358 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:49.888367 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:49.888381 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:49.888417 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:49.891076 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:49.891773 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:49.891779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:49.891785 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:49.891942 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:49.892497 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:49.892502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:49.892512 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:49.893098 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:49.893423 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:49.893908 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 10:36:49.894395 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":267,"threads_started":1}
I20260925 10:36:49.896132 17431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:36:49.978972 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 17308
2026-09-25T10:36:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64250 ms)
[----------] 30 tests from SecurityITest (294308 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:36:50 dist-test-slave-dxdd krb5kdc[17493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:36:50 dist-test-slave-dxdd krb5kdc[17493](info): set up 2 sockets
Sep 25 10:36:50 dist-test-slave-dxdd krb5kdc[17493](info): commencing operation
krb5kdc: starting...
W20260925 10:36:51.082267 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:36:51 dist-test-slave-dxdd krb5kdc[17493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:36:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:36:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:36:51.303288 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39323
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:36:51.614418 17506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:51.614682 17506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:51.614861 17506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:51.622573 17506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:36:51.622686 17506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:51.622767 17506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:51.622884 17506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:36:51.622969 17506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:36:51.633795 17506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:51.635170 17506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:51.636430 17506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:51.648120 17511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:51.648152 17512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:51.648093 17514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:51.648556 17506 server_base.cc:1034] running on GCE node
I20260925 10:36:51.649360 17506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:51.650734 17506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:51.651958 17506 hybrid_clock.cc:648] HybridClock initialized: now 1790332611651929 us; error 41 us; skew 500 ppm
I20260925 10:36:51.652283 17506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:36:51 dist-test-slave-dxdd krb5kdc[17493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:36:51.655377 17506 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:36:51.656793 17506 webserver.cc:460] Webserver started at http://127.0.97.62:40437/ using document root <none> and password file <none>
I20260925 10:36:51.657449 17506 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:51.657589 17506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:51.657891 17506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:36:51.660682 17506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "6700dc4d296c4837b6cae43d012b9f70"
format_stamp: "Formatted at 2026-09-25 10:36:51 on dist-test-slave-dxdd"
server_key: "dbcb88861fa9cef2e5572ce199c3ea96"
server_key_iv: "16d25907f7cc76e4d1615eabde3b1cf7"
server_key_version: "encryptionkey@0"
I20260925 10:36:51.661484 17506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "6700dc4d296c4837b6cae43d012b9f70"
format_stamp: "Formatted at 2026-09-25 10:36:51 on dist-test-slave-dxdd"
server_key: "dbcb88861fa9cef2e5572ce199c3ea96"
server_key_iv: "16d25907f7cc76e4d1615eabde3b1cf7"
server_key_version: "encryptionkey@0"
I20260925 10:36:51.666240 17506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 10:36:51.669543 17521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:51.670775 17506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 10:36:51.670944 17506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "6700dc4d296c4837b6cae43d012b9f70"
format_stamp: "Formatted at 2026-09-25 10:36:51 on dist-test-slave-dxdd"
server_key: "dbcb88861fa9cef2e5572ce199c3ea96"
server_key_iv: "16d25907f7cc76e4d1615eabde3b1cf7"
server_key_version: "encryptionkey@0"
I20260925 10:36:51.671129 17506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:51.686291 17506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:51.689797 17506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:51.690065 17506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:51.707621 17506 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39323
I20260925 10:36:51.707657 17572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39323 every 8 connection(s)
I20260925 10:36:51.708910 17506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:36:51.712394 17573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:36:51.714805 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 17506
I20260925 10:36:51.715011 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:36:51.715646 388 external_mini_cluster.cc:1428] Setting key f1e1a2ac3583e4d8cf7d06cbb3e9c0bc
I20260925 10:36:51.723261 17573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70: Bootstrap starting.
I20260925 10:36:51.726047 17573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70: Neither blocks nor log segments found. Creating new log.
I20260925 10:36:51.727049 17573 log.cc:824] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:36:51 dist-test-slave-dxdd krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:36:51.729892 17573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70: No bootstrap required, opened a new log
I20260925 10:36:51.737246 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:51.717471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.0.97.62:39323)
0925 10:36:51.718273 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:51.718283 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:51.718386 (+ 103us) server_negotiation.cc:407] Connection header received
0925 10:36:51.720075 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:51.720096 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:51.720485 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:51.720860 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:51.722237 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:51.723650 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:51.725098 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:51.725738 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:51.725932 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:51.730529 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:51.730536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:51.730549 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:51.730583 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:51.733407 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:51.734225 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:51.734231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:51.734236 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:51.734450 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:51.734823 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:51.734825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:51.734827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:51.735307 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:51.735511 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:51.735924 (+ 413us) server_negotiation.cc:299] Negotiation successful
0925 10:36:51.736285 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":332,"threads_started":1}
I20260925 10:36:51.738111 17573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6700dc4d296c4837b6cae43d012b9f70" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39323 } }
I20260925 10:36:51.738505 17573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:36:51.738616 17573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6700dc4d296c4837b6cae43d012b9f70, State: Initialized, Role: FOLLOWER
I20260925 10:36:51.739187 17573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6700dc4d296c4837b6cae43d012b9f70" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39323 } }
I20260925 10:36:51.739470 17573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:36:51.739619 17573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:36:51.739796 17573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:36:51.741863 17573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6700dc4d296c4837b6cae43d012b9f70" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39323 } }
I20260925 10:36:51.742373 17573 leader_election.cc:302] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6700dc4d296c4837b6cae43d012b9f70; no voters:
I20260925 10:36:51.742909 17573 leader_election.cc:288] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:36:51.743010 17578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:36:51.743708 17578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [term 1 LEADER]: Becoming Leader. State: Replica: 6700dc4d296c4837b6cae43d012b9f70, State: Running, Role: LEADER
I20260925 10:36:51.744272 17578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6700dc4d296c4837b6cae43d012b9f70" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39323 } }
I20260925 10:36:51.744889 17573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:36:51.746984 17579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6700dc4d296c4837b6cae43d012b9f70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6700dc4d296c4837b6cae43d012b9f70" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39323 } } }
I20260925 10:36:51.747109 17580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6700dc4d296c4837b6cae43d012b9f70. Latest consensus state: current_term: 1 leader_uuid: "6700dc4d296c4837b6cae43d012b9f70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6700dc4d296c4837b6cae43d012b9f70" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39323 } } }
I20260925 10:36:51.747367 17579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [sys.catalog]: This master's current role is: LEADER
I20260925 10:36:51.747368 17580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70 [sys.catalog]: This master's current role is: LEADER
I20260925 10:36:51.748844 17584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:36:51.754027 17584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:36:51.761925 17584 catalog_manager.cc:1347] Generated new cluster ID: b442d3dc1df04e698f3371c3a443bf8e
I20260925 10:36:51.762030 17584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:36:51.787699 17584 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:36:51.788954 17584 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:36:51.816708 17584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70: Generated new TSK 0
I20260925 10:36:51.817504 17584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:36:51.886548 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39323
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:36:52.205607 17599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:52.205907 17599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:52.206099 17599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:52.212989 17599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:52.213131 17599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:52.213311 17599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:36:52.222559 17599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:52.223733 17599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:52.224927 17599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:52.237120 17607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:52.237339 17604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:52.237638 17605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:52.238528 17599 server_base.cc:1034] running on GCE node
I20260925 10:36:52.239099 17599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:52.239938 17599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:52.241153 17599 hybrid_clock.cc:648] HybridClock initialized: now 1790332612241114 us; error 55 us; skew 500 ppm
I20260925 10:36:52.241457 17599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:36:52 dist-test-slave-dxdd krb5kdc[17493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:36:52.244653 17599 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:36:52.246063 17599 webserver.cc:460] Webserver started at http://127.0.97.1:33967/ using document root <none> and password file <none>
I20260925 10:36:52.246717 17599 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:52.246812 17599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:52.247042 17599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:36:52.249753 17599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "30c988d404314a6daae06214cd6b2a60"
format_stamp: "Formatted at 2026-09-25 10:36:52 on dist-test-slave-dxdd"
server_key: "26e1f18ef410a204d4baccbf9e5d1999"
server_key_iv: "3ae02d373ef96504611ddd26bcda8435"
server_key_version: "encryptionkey@0"
I20260925 10:36:52.250559 17599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "30c988d404314a6daae06214cd6b2a60"
format_stamp: "Formatted at 2026-09-25 10:36:52 on dist-test-slave-dxdd"
server_key: "26e1f18ef410a204d4baccbf9e5d1999"
server_key_iv: "3ae02d373ef96504611ddd26bcda8435"
server_key_version: "encryptionkey@0"
I20260925 10:36:52.255297 17599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:36:52.258445 17614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:52.259668 17599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:36:52.259835 17599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "30c988d404314a6daae06214cd6b2a60"
format_stamp: "Formatted at 2026-09-25 10:36:52 on dist-test-slave-dxdd"
server_key: "26e1f18ef410a204d4baccbf9e5d1999"
server_key_iv: "3ae02d373ef96504611ddd26bcda8435"
server_key_version: "encryptionkey@0"
I20260925 10:36:52.260007 17599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:52.285452 17599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:52.289088 17599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:52.289350 17599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:52.290221 17599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:36:52.291445 17599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:36:52.291514 17599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:52.291663 17599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:36:52.291720 17599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:52.324245 17599 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32933
I20260925 10:36:52.324291 17726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32933 every 8 connection(s)
I20260925 10:36:52.325508 17599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:36:52.329885 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 17599
I20260925 10:36:52.330094 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:36:52.330749 388 external_mini_cluster.cc:1428] Setting key 0ccbdba4de3a882efe90e695b47733b3
Sep 25 10:36:52 dist-test-slave-dxdd krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:36:52.346160 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:52.327985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47185 (local address 127.0.97.62:39323)
0925 10:36:52.328453 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:52.328460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:52.329349 (+ 889us) server_negotiation.cc:407] Connection header received
0925 10:36:52.331311 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:52.331316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:52.331410 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:52.331538 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:52.333286 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:52.334531 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:52.335965 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:52.336617 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:52.336754 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:52.340234 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:52.340242 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:52.340250 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:52.340293 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:52.342968 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:52.343860 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:52.343863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:52.343866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:52.343981 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:52.344441 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:52.344445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:52.344468 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:52.344727 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:52.344848 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:52.345693 (+ 845us) server_negotiation.cc:299] Negotiation successful
0925 10:36:52.345850 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":219,"threads_started":1}
I20260925 10:36:52.347616 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:52.328514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39323 (local address 127.0.97.1:47185)
0925 10:36:52.329197 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:52.329231 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:52.331057 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:52.331782 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:52.331793 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:52.332492 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:36:52.333120 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:52.333137 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:52.334673 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:52.334676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:52.335823 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:52.335838 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:52.336806 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:52.336810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:52.336994 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:52.337669 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:36:52.337689 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:36:52.340022 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:36:52.343098 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:36:52.343109 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:36:52.343127 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:36:52.343750 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:36:52.344087 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:36:52.344094 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:36:52.344100 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:36:52.344334 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:36:52.344868 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:36:52.344878 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:36:52.345314 (+ 436us) client_negotiation.cc:769] Sending connection context
0925 10:36:52.345602 (+ 288us) client_negotiation.cc:240] Negotiation successful
0925 10:36:52.346176 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":269,"threads_started":1}
I20260925 10:36:52.349234 17727 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39323
I20260925 10:36:52.349529 17727 heartbeater.cc:464] Registering TS with master...
I20260925 10:36:52.350379 17727 heartbeater.cc:511] Master 127.0.97.62:39323 requested a full tablet report, sending...
I20260925 10:36:52.352356 17538 ts_manager.cc:194] Registered new tserver with Master: 30c988d404314a6daae06214cd6b2a60 (127.0.97.1:32933)
I20260925 10:36:52.353734 17538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47185
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:36:52.385651 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39323
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:36:52.703698 17733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:52.703958 17733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:52.704051 17733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:52.711238 17733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:52.711351 17733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:52.711544 17733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:36:52.720847 17733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:52.721980 17733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:52.723348 17733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:52.736079 17739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:52.736238 17741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:52.736263 17738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:52.737154 17733 server_base.cc:1034] running on GCE node
I20260925 10:36:52.737713 17733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:52.738631 17733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:52.739825 17733 hybrid_clock.cc:648] HybridClock initialized: now 1790332612739796 us; error 30 us; skew 500 ppm
I20260925 10:36:52.740072 17733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:36:52 dist-test-slave-dxdd krb5kdc[17493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:36:52.742880 17733 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:36:52.744129 17733 webserver.cc:460] Webserver started at http://127.0.97.2:43579/ using document root <none> and password file <none>
I20260925 10:36:52.744719 17733 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:52.744830 17733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:52.745100 17733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:36:52.747753 17733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "b13095b1bd71450c809d867430586fa4"
format_stamp: "Formatted at 2026-09-25 10:36:52 on dist-test-slave-dxdd"
server_key: "c7ee761c5c2806741224725ffcffd794"
server_key_iv: "de28fe2a64d50e16a0bd0601576b0d28"
server_key_version: "encryptionkey@0"
I20260925 10:36:52.748458 17733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b13095b1bd71450c809d867430586fa4"
format_stamp: "Formatted at 2026-09-25 10:36:52 on dist-test-slave-dxdd"
server_key: "c7ee761c5c2806741224725ffcffd794"
server_key_iv: "de28fe2a64d50e16a0bd0601576b0d28"
server_key_version: "encryptionkey@0"
I20260925 10:36:52.753314 17733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:36:52.756677 17748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:52.757949 17733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:36:52.758113 17733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "b13095b1bd71450c809d867430586fa4"
format_stamp: "Formatted at 2026-09-25 10:36:52 on dist-test-slave-dxdd"
server_key: "c7ee761c5c2806741224725ffcffd794"
server_key_iv: "de28fe2a64d50e16a0bd0601576b0d28"
server_key_version: "encryptionkey@0"
I20260925 10:36:52.758313 17733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:52.775818 17733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:52.779994 17733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:52.780272 17733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:52.781190 17733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:36:52.782465 17733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:36:52.782564 17733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:52.782760 17733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:36:52.782841 17733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:52.814991 17733 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37853
I20260925 10:36:52.815073 17860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37853 every 8 connection(s)
I20260925 10:36:52.816191 17733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:36:52.817492 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 17733
I20260925 10:36:52.817668 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:36:52.818210 388 external_mini_cluster.cc:1428] Setting key edc45c3676022c5e380e5875d6d5fdbe
Sep 25 10:36:52 dist-test-slave-dxdd krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:36:52.838531 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:52.819458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33629 (local address 127.0.97.62:39323)
0925 10:36:52.819643 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:52.819648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:52.820934 (+ 1286us) server_negotiation.cc:407] Connection header received
0925 10:36:52.823772 (+ 2838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:52.823776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:52.823840 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:52.823922 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:52.825857 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:52.826594 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:52.829084 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:52.829512 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:52.829658 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:52.833054 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:52.833059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:52.833063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:52.833093 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:52.835644 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:52.836312 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:52.836317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:52.836321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:52.836449 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:52.836861 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:52.836866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:52.836869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:52.837117 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:52.837293 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:52.837922 (+ 629us) server_negotiation.cc:299] Negotiation successful
0925 10:36:52.838149 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:36:52.839526 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:52.819922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39323 (local address 127.0.97.2:33629)
0925 10:36:52.820749 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:52.820804 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:52.823492 (+ 2688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:52.824175 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:52.824188 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:52.824931 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:36:52.825696 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:52.825722 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:52.826744 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:52.826751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:52.828873 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:52.828890 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:52.829745 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:52.829751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:52.829979 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:52.830688 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:36:52.830709 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:36:52.832847 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:36:52.835779 (+ 2932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:36:52.835788 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:36:52.835802 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:36:52.836194 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:36:52.836556 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:36:52.836562 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:36:52.836566 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:36:52.836739 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:36:52.837241 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:36:52.837248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:36:52.837578 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 10:36:52.837839 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 10:36:52.838436 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":357,"threads_started":1}
I20260925 10:36:52.841014 17861 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39323
I20260925 10:36:52.841349 17861 heartbeater.cc:464] Registering TS with master...
I20260925 10:36:52.842238 17861 heartbeater.cc:511] Master 127.0.97.62:39323 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260925 10:36:52.843757 17538 ts_manager.cc:194] Registered new tserver with Master: b13095b1bd71450c809d867430586fa4 (127.0.97.2:37853)
I20260925 10:36:52.844662 17538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:36:52.871774 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39323
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:36:53.184237 17866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:53.184499 17866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:53.184674 17866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:53.191751 17866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:53.191870 17866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:53.192049 17866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:36:53.201376 17866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:53.202653 17866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:53.203898 17866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:53.216598 17871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:53.216573 17874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:53.216661 17872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:53.222643 17866 server_base.cc:1034] running on GCE node
I20260925 10:36:53.223234 17866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:53.224136 17866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:53.225360 17866 hybrid_clock.cc:648] HybridClock initialized: now 1790332613225328 us; error 56 us; skew 500 ppm
I20260925 10:36:53.225667 17866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:36:53 dist-test-slave-dxdd krb5kdc[17493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:36:53.228626 17866 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:36:53.230002 17866 webserver.cc:460] Webserver started at http://127.0.97.3:42755/ using document root <none> and password file <none>
I20260925 10:36:53.230754 17866 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:53.230896 17866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:53.231192 17866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:36:53.233955 17866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "fc5a2457b3bb47a29dfc5ff79fa71a0c"
format_stamp: "Formatted at 2026-09-25 10:36:53 on dist-test-slave-dxdd"
server_key: "a4cb4ffc53c963ccb3033fa1da6a055d"
server_key_iv: "d4c45348c8bd84c395908d37ba581dc0"
server_key_version: "encryptionkey@0"
I20260925 10:36:53.234723 17866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fc5a2457b3bb47a29dfc5ff79fa71a0c"
format_stamp: "Formatted at 2026-09-25 10:36:53 on dist-test-slave-dxdd"
server_key: "a4cb4ffc53c963ccb3033fa1da6a055d"
server_key_iv: "d4c45348c8bd84c395908d37ba581dc0"
server_key_version: "encryptionkey@0"
I20260925 10:36:53.239409 17866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:36:53.242626 17881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:53.243817 17866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:36:53.243983 17866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "fc5a2457b3bb47a29dfc5ff79fa71a0c"
format_stamp: "Formatted at 2026-09-25 10:36:53 on dist-test-slave-dxdd"
server_key: "a4cb4ffc53c963ccb3033fa1da6a055d"
server_key_iv: "d4c45348c8bd84c395908d37ba581dc0"
server_key_version: "encryptionkey@0"
I20260925 10:36:53.244179 17866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:53.260584 17866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:53.264765 17866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:53.264987 17866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:53.265823 17866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:36:53.267086 17866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:36:53.267156 17866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:53.267259 17866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:36:53.267305 17866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:53.298929 17866 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46047
I20260925 10:36:53.298974 17993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46047 every 8 connection(s)
I20260925 10:36:53.300168 17866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:36:53.302696 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 17866
I20260925 10:36:53.302891 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:36:53.303328 388 external_mini_cluster.cc:1428] Setting key 8ee165d679e349e69929158bf0402f77
Sep 25 10:36:53 dist-test-slave-dxdd krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:36:53.319716 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.302561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42911 (local address 127.0.97.62:39323)
0925 10:36:53.302767 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:53.302773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:53.304094 (+ 1321us) server_negotiation.cc:407] Connection header received
0925 10:36:53.306549 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:53.306554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:53.306633 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:53.306752 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:53.308177 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.308964 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.310536 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.310948 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.311080 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.314274 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:53.314280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:53.314285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:53.314339 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:53.316691 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:53.317523 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:53.317528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:53.317531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:53.317690 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:53.318182 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:53.318185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:53.318187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:53.318606 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:53.318761 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:53.319323 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 10:36:53.319487 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:36:53.320624 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.303116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39323 (local address 127.0.97.3:42911)
0925 10:36:53.303945 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:53.303981 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:53.306221 (+ 2240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:53.306975 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:53.306983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:53.307468 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:36:53.307984 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:53.308000 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.309113 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.309119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:53.310404 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:53.310415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.311163 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.311171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:53.311423 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.312082 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:36:53.312102 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:36:53.314079 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:36:53.316837 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:36:53.316846 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:36:53.316860 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:36:53.317395 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:36:53.317827 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:36:53.317831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:36:53.317834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:36:53.318071 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:36:53.318734 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:36:53.318739 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:36:53.319014 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 10:36:53.319230 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 10:36:53.319561 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":324,"threads_started":1}
I20260925 10:36:53.321877 17994 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39323
I20260925 10:36:53.322124 17994 heartbeater.cc:464] Registering TS with master...
I20260925 10:36:53.322834 17994 heartbeater.cc:511] Master 127.0.97.62:39323 requested a full tablet report, sending...
I20260925 10:36:53.324103 17538 ts_manager.cc:194] Registered new tserver with Master: fc5a2457b3bb47a29dfc5ff79fa71a0c (127.0.97.3:46047)
I20260925 10:36:53.324898 17538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42911
I20260925 10:36:53.330659 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:36:53.347136 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.334992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.0.97.62:39323)
0925 10:36:53.335145 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:53.335149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:53.335606 (+ 457us) server_negotiation.cc:407] Connection header received
0925 10:36:53.335793 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:53.335798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:53.335883 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:53.335979 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:53.337442 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.338123 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.339431 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.339827 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.339950 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.342141 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:53.342147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:53.342151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:53.342188 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:53.344557 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:53.345220 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:53.345227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:53.345233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:53.345357 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:53.345716 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:53.345729 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:53.345732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:53.346012 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:53.346256 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:53.346603 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 10:36:53.346827 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:36:53.351039 17538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 10:36:53.356622 17727 heartbeater.cc:503] Master 127.0.97.62:39323 was elected leader, sending a full tablet report...
W20260925 10:36:53.357764 17538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:36:53.382714 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.368735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32933 (local address 127.0.0.1:47156)
0925 10:36:53.370373 (+ 1638us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:53.370408 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:53.370550 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:53.374535 (+ 3985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:53.374542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:53.374570 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:36:53.375006 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:53.375019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.377200 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.377206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:53.379720 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:53.379744 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.381615 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.381622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:53.381834 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.381883 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 10:36:53.381987 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:36:53.382184 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1357,"mutex_wait_us":2,"thread_start_us":4958,"threads_started":1}
I20260925 10:36:53.383205 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.369930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.0.97.3:46047)
0925 10:36:53.373545 (+ 3615us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:53.373555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:53.374121 (+ 566us) server_negotiation.cc:407] Connection header received
0925 10:36:53.374436 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:53.374445 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:53.374658 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:53.374815 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:36:53.376149 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.377737 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.380325 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.381720 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.381816 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.382045 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:53.382680 (+ 635us) server_negotiation.cc:299] Negotiation successful
0925 10:36:53.382848 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3435,"thread_start_us":3279,"threads_started":1}
I20260925 10:36:53.384104 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.369368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46047 (local address 127.0.0.1:44838)
0925 10:36:53.373994 (+ 4626us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:53.374010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:53.374215 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:53.375129 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:53.375180 (+ 51us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:53.375205 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:36:53.375933 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:53.375948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.377920 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.377925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:53.380141 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:53.380154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.382237 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.382243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:53.382453 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.382481 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 10:36:53.383576 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0925 10:36:53.383719 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4459,"mutex_wait_us":78,"thread_start_us":833,"threads_started":1}
I20260925 10:36:53.384263 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.369157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.0.97.1:32933)
0925 10:36:53.373911 (+ 4754us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:53.373919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:53.373947 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:36:53.374044 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:53.374053 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:53.374325 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:53.375196 (+ 871us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:36:53.375413 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.377035 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.379905 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.381392 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.383455 (+ 2063us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.383694 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:53.383769 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 10:36:53.383926 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4562,"thread_start_us":4371,"threads_started":1}
I20260925 10:36:53.387823 17662 tablet_service.cc:1467] Processing CreateTablet for tablet c94d3e4a065d4f11aecf1416d38e6ef0 (DEFAULT_TABLE table=test-table [id=cbdb0488033a461d9800114550bb8042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:36:53.387868 17929 tablet_service.cc:1467] Processing CreateTablet for tablet c94d3e4a065d4f11aecf1416d38e6ef0 (DEFAULT_TABLE table=test-table [id=cbdb0488033a461d9800114550bb8042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:36:53.389077 17662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94d3e4a065d4f11aecf1416d38e6ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:36:53.389207 17929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94d3e4a065d4f11aecf1416d38e6ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:36:53.391687 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.374365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52536 (local address 127.0.97.2:37853)
0925 10:36:53.384550 (+ 10185us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:53.384559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:53.384591 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:36:53.384696 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:53.384704 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:53.384936 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:53.385135 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:36:53.385515 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.387064 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.389182 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.390629 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.390766 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.390965 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:53.391187 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 10:36:53.391359 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10036,"thread_start_us":9898,"threads_started":1}
I20260925 10:36:53.392666 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.374075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37853 (local address 127.0.0.1:52536)
0925 10:36:53.382926 (+ 8851us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:53.382947 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:53.383036 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:53.385137 (+ 2101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:53.385142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:53.385168 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:36:53.385337 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:53.385348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.387228 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.387233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:53.389014 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:53.389027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.390805 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.390813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:53.390993 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.391025 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:36:53.392176 (+ 1151us) client_negotiation.cc:240] Negotiation successful
0925 10:36:53.392369 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8765,"thread_start_us":9606,"threads_started":1}
I20260925 10:36:53.396523 17796 tablet_service.cc:1467] Processing CreateTablet for tablet c94d3e4a065d4f11aecf1416d38e6ef0 (DEFAULT_TABLE table=test-table [id=cbdb0488033a461d9800114550bb8042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:36:53.397954 17796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94d3e4a065d4f11aecf1416d38e6ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:36:53.401001 18012 tablet_bootstrap.cc:492] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c: Bootstrap starting.
I20260925 10:36:53.403029 18013 tablet_bootstrap.cc:492] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60: Bootstrap starting.
I20260925 10:36:53.404625 18012 tablet_bootstrap.cc:654] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c: Neither blocks nor log segments found. Creating new log.
I20260925 10:36:53.405867 18012 log.cc:824] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:36:53.406569 18013 tablet_bootstrap.cc:654] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60: Neither blocks nor log segments found. Creating new log.
I20260925 10:36:53.407817 18013 log.cc:824] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60: Log is configured to *not* fsync() on all Append() calls
I20260925 10:36:53.408850 18012 tablet_bootstrap.cc:492] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c: No bootstrap required, opened a new log
I20260925 10:36:53.409103 18012 ts_tablet_manager.cc:1397] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260925 10:36:53.410413 18013 tablet_bootstrap.cc:492] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60: No bootstrap required, opened a new log
I20260925 10:36:53.410727 18013 ts_tablet_manager.cc:1397] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260925 10:36:53.411749 18016 tablet_bootstrap.cc:492] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4: Bootstrap starting.
I20260925 10:36:53.415366 18016 tablet_bootstrap.cc:654] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4: Neither blocks nor log segments found. Creating new log.
I20260925 10:36:53.416491 18016 log.cc:824] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4: Log is configured to *not* fsync() on all Append() calls
I20260925 10:36:53.417127 18012 raft_consensus.cc:348] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c988d404314a6daae06214cd6b2a60" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32933 } } peers { permanent_uuid: "b13095b1bd71450c809d867430586fa4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37853 } } peers { permanent_uuid: "fc5a2457b3bb47a29dfc5ff79fa71a0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46047 } }
I20260925 10:36:53.417591 18012 raft_consensus.cc:374] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:36:53.417737 18012 raft_consensus.cc:729] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5a2457b3bb47a29dfc5ff79fa71a0c, State: Initialized, Role: FOLLOWER
I20260925 10:36:53.418440 18012 consensus_queue.cc:260] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c988d404314a6daae06214cd6b2a60" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32933 } } peers { permanent_uuid: "b13095b1bd71450c809d867430586fa4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37853 } } peers { permanent_uuid: "fc5a2457b3bb47a29dfc5ff79fa71a0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46047 } }
I20260925 10:36:53.419488 18016 tablet_bootstrap.cc:492] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4: No bootstrap required, opened a new log
I20260925 10:36:53.419559 17994 heartbeater.cc:503] Master 127.0.97.62:39323 was elected leader, sending a full tablet report...
I20260925 10:36:53.419755 18016 ts_tablet_manager.cc:1397] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 10:36:53.420164 18012 ts_tablet_manager.cc:1428] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260925 10:36:53.420948 18013 raft_consensus.cc:348] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c988d404314a6daae06214cd6b2a60" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32933 } } peers { permanent_uuid: "b13095b1bd71450c809d867430586fa4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37853 } } peers { permanent_uuid: "fc5a2457b3bb47a29dfc5ff79fa71a0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46047 } }
I20260925 10:36:53.421310 18013 raft_consensus.cc:374] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:36:53.421423 18013 raft_consensus.cc:729] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c988d404314a6daae06214cd6b2a60, State: Initialized, Role: FOLLOWER
I20260925 10:36:53.421955 18013 consensus_queue.cc:260] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c988d404314a6daae06214cd6b2a60" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32933 } } peers { permanent_uuid: "b13095b1bd71450c809d867430586fa4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37853 } } peers { permanent_uuid: "fc5a2457b3bb47a29dfc5ff79fa71a0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46047 } }
I20260925 10:36:53.423602 18013 ts_tablet_manager.cc:1428] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
I20260925 10:36:53.428433 18016 raft_consensus.cc:348] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c988d404314a6daae06214cd6b2a60" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32933 } } peers { permanent_uuid: "b13095b1bd71450c809d867430586fa4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37853 } } peers { permanent_uuid: "fc5a2457b3bb47a29dfc5ff79fa71a0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46047 } }
I20260925 10:36:53.428750 18016 raft_consensus.cc:374] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:36:53.428890 18016 raft_consensus.cc:729] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13095b1bd71450c809d867430586fa4, State: Initialized, Role: FOLLOWER
I20260925 10:36:53.429392 18016 consensus_queue.cc:260] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c988d404314a6daae06214cd6b2a60" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32933 } } peers { permanent_uuid: "b13095b1bd71450c809d867430586fa4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37853 } } peers { permanent_uuid: "fc5a2457b3bb47a29dfc5ff79fa71a0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46047 } }
I20260925 10:36:53.430402 17861 heartbeater.cc:503] Master 127.0.97.62:39323 was elected leader, sending a full tablet report...
I20260925 10:36:53.430871 18016 ts_tablet_manager.cc:1428] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260925 10:36:53.486645 18019 raft_consensus.cc:482] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:36:53.486922 18019 raft_consensus.cc:504] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c988d404314a6daae06214cd6b2a60" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32933 } } peers { permanent_uuid: "b13095b1bd71450c809d867430586fa4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37853 } } peers { permanent_uuid: "fc5a2457b3bb47a29dfc5ff79fa71a0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46047 } }
I20260925 10:36:53.488692 18019 leader_election.cc:288] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b13095b1bd71450c809d867430586fa4 (127.0.97.2:37853), fc5a2457b3bb47a29dfc5ff79fa71a0c (127.0.97.3:46047)
I20260925 10:36:53.497839 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.489043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39417 (local address 127.0.97.2:37853)
0925 10:36:53.489277 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:53.489285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:53.490111 (+ 826us) server_negotiation.cc:407] Connection header received
0925 10:36:53.490329 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:53.490337 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:53.490415 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:53.490517 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:36:53.491321 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.492408 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.495376 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.496596 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.496732 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.496827 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:53.497457 (+ 630us) server_negotiation.cc:299] Negotiation successful
0925 10:36:53.497584 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:36:53.497839 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.489252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40687 (local address 127.0.97.3:46047)
0925 10:36:53.489468 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:53.489475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:53.489929 (+ 454us) server_negotiation.cc:407] Connection header received
0925 10:36:53.490281 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:53.490287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:53.490392 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:53.490455 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:36:53.491322 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.492407 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.495376 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.496599 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.496738 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.496830 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:53.497481 (+ 651us) server_negotiation.cc:299] Negotiation successful
0925 10:36:53.497600 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:36:53.498024 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.489062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37853 (local address 127.0.97.1:39417)
0925 10:36:53.490007 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:53.490029 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:53.490157 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:53.490538 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:53.490542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:53.490568 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:36:53.491108 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:53.491123 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.492540 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.492545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:53.495216 (+ 2671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:53.495230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.496854 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.496865 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:53.497181 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.497299 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 10:36:53.497399 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 10:36:53.497537 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":303,"thread_start_us":436,"threads_started":1}
I20260925 10:36:53.498420 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.489040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46047 (local address 127.0.97.1:40687)
0925 10:36:53.489822 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:53.489841 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:53.490077 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:53.490658 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:53.490664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:53.490699 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:36:53.491181 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:53.491190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.492542 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.492546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:53.495217 (+ 2671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:53.495230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.496876 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.496881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:53.497183 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.497297 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 10:36:53.497437 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 10:36:53.497536 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":298,"thread_start_us":744,"threads_started":1}
I20260925 10:36:53.499131 17816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94d3e4a065d4f11aecf1416d38e6ef0" candidate_uuid: "30c988d404314a6daae06214cd6b2a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b13095b1bd71450c809d867430586fa4" is_pre_election: true
I20260925 10:36:53.499356 17949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94d3e4a065d4f11aecf1416d38e6ef0" candidate_uuid: "30c988d404314a6daae06214cd6b2a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc5a2457b3bb47a29dfc5ff79fa71a0c" is_pre_election: true
I20260925 10:36:53.499739 17816 raft_consensus.cc:2454] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c988d404314a6daae06214cd6b2a60 in term 0.
I20260925 10:36:53.499845 17949 raft_consensus.cc:2454] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c988d404314a6daae06214cd6b2a60 in term 0.
I20260925 10:36:53.500470 17617 leader_election.cc:302] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c988d404314a6daae06214cd6b2a60, b13095b1bd71450c809d867430586fa4; no voters:
I20260925 10:36:53.500877 18019 raft_consensus.cc:2781] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:36:53.501030 18019 raft_consensus.cc:482] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:36:53.501134 18019 raft_consensus.cc:3037] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:36:53.503399 18019 raft_consensus.cc:504] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c988d404314a6daae06214cd6b2a60" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32933 } } peers { permanent_uuid: "b13095b1bd71450c809d867430586fa4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37853 } } peers { permanent_uuid: "fc5a2457b3bb47a29dfc5ff79fa71a0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46047 } }
I20260925 10:36:53.503978 18019 leader_election.cc:288] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [CANDIDATE]: Term 1 election: Requested vote from peers b13095b1bd71450c809d867430586fa4 (127.0.97.2:37853), fc5a2457b3bb47a29dfc5ff79fa71a0c (127.0.97.3:46047)
I20260925 10:36:53.504419 17816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94d3e4a065d4f11aecf1416d38e6ef0" candidate_uuid: "30c988d404314a6daae06214cd6b2a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b13095b1bd71450c809d867430586fa4"
I20260925 10:36:53.504434 17949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94d3e4a065d4f11aecf1416d38e6ef0" candidate_uuid: "30c988d404314a6daae06214cd6b2a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc5a2457b3bb47a29dfc5ff79fa71a0c"
I20260925 10:36:53.504614 17816 raft_consensus.cc:3037] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:36:53.504621 17949 raft_consensus.cc:3037] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:36:53.506839 17816 raft_consensus.cc:2454] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c988d404314a6daae06214cd6b2a60 in term 1.
I20260925 10:36:53.506867 17949 raft_consensus.cc:2454] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c988d404314a6daae06214cd6b2a60 in term 1.
I20260925 10:36:53.507243 17617 leader_election.cc:302] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c988d404314a6daae06214cd6b2a60, b13095b1bd71450c809d867430586fa4; no voters:
I20260925 10:36:53.507519 18019 raft_consensus.cc:2781] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:36:53.507925 18019 raft_consensus.cc:686] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [term 1 LEADER]: Becoming Leader. State: Replica: 30c988d404314a6daae06214cd6b2a60, State: Running, Role: LEADER
I20260925 10:36:53.508379 18019 consensus_queue.cc:237] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c988d404314a6daae06214cd6b2a60" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32933 } } peers { permanent_uuid: "b13095b1bd71450c809d867430586fa4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37853 } } peers { permanent_uuid: "fc5a2457b3bb47a29dfc5ff79fa71a0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46047 } }
I20260925 10:36:53.512882 17537 catalog_manager.cc:5495] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c988d404314a6daae06214cd6b2a60 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "30c988d404314a6daae06214cd6b2a60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c988d404314a6daae06214cd6b2a60" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b13095b1bd71450c809d867430586fa4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc5a2457b3bb47a29dfc5ff79fa71a0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46047 } health_report { overall_health: UNKNOWN } } }
I20260925 10:36:53.531857 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.526428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.0.97.1:32933)
0925 10:36:53.526640 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:53.526646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:53.526746 (+ 100us) server_negotiation.cc:407] Connection header received
0925 10:36:53.526984 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:53.526988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:53.527059 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:53.527205 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:36:53.527571 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.528258 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.529758 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.530179 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.530318 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.530802 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:36:53.531296 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:36:53.531451 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:53.531525 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 10:36:53.531603 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:36:53.539615 17816 raft_consensus.cc:1261] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4 [term 1 FOLLOWER]: Refusing update from remote peer 30c988d404314a6daae06214cd6b2a60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:36:53.539615 17949 raft_consensus.cc:1261] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c [term 1 FOLLOWER]: Refusing update from remote peer 30c988d404314a6daae06214cd6b2a60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:36:53.540550 18023 consensus_queue.cc:1035] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5a2457b3bb47a29dfc5ff79fa71a0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:36:53.540865 18019 consensus_queue.cc:1035] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b13095b1bd71450c809d867430586fa4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:36:53.551736 17995 tablet.cc:2369] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:36:53.556442 18028 mvcc.cc:205] Tried to move back new op lower bound from 7333202385049112576 to 7333202384933855232. Current Snapshot: MvccSnapshot[applied={T|T < 7333202385049112576}]
I20260925 10:36:53.566294 17537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:36:53.566615 17537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:36:53.569732 17537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6700dc4d296c4837b6cae43d012b9f70: Sending DeleteTablet for 3 replicas of tablet c94d3e4a065d4f11aecf1416d38e6ef0
I20260925 10:36:53.570725 17662 tablet_service.cc:1514] Processing DeleteTablet for tablet c94d3e4a065d4f11aecf1416d38e6ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:36:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47156
I20260925 10:36:53.570888 17796 tablet_service.cc:1514] Processing DeleteTablet for tablet c94d3e4a065d4f11aecf1416d38e6ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:36:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52536
I20260925 10:36:53.571079 17929 tablet_service.cc:1514] Processing DeleteTablet for tablet c94d3e4a065d4f11aecf1416d38e6ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:36:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44838
I20260925 10:36:53.583191 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 17599
I20260925 10:36:53.613724 18038 tablet_replica.cc:330] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c: stopping tablet replica
I20260925 10:36:53.617337 18038 raft_consensus.cc:2229] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:36:53.618290 18038 raft_consensus.cc:2258] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:36:53.622938 18036 tablet_replica.cc:330] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60: stopping tablet replica
I20260925 10:36:53.627218 18036 raft_consensus.cc:2229] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:36:53.628273 18036 raft_consensus.cc:2258] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:36:53.624922 18038 ts_tablet_manager.cc:1905] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:36:53.636031 18036 ts_tablet_manager.cc:1905] T c94d3e4a065d4f11aecf1416d38e6ef0 P 30c988d404314a6daae06214cd6b2a60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:36:53.636787 18037 tablet_replica.cc:330] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4: stopping tablet replica
I20260925 10:36:53.638780 18037 raft_consensus.cc:2229] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:36:53.639339 18037 raft_consensus.cc:2258] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 25 10:36:53 dist-test-slave-dxdd krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:36:53.641582 18037 ts_tablet_manager.cc:1905] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:36:53.648561 18038 ts_tablet_manager.cc:1918] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:36:53.648607 18037 ts_tablet_manager.cc:1918] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:36:53.648751 18037 log.cc:1196] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/c94d3e4a065d4f11aecf1416d38e6ef0
I20260925 10:36:53.648885 18038 log.cc:1196] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/c94d3e4a065d4f11aecf1416d38e6ef0
I20260925 10:36:53.649382 18037 ts_tablet_manager.cc:1939] T c94d3e4a065d4f11aecf1416d38e6ef0 P b13095b1bd71450c809d867430586fa4: Deleting consensus metadata
I20260925 10:36:53.649616 18038 ts_tablet_manager.cc:1939] T c94d3e4a065d4f11aecf1416d38e6ef0 P fc5a2457b3bb47a29dfc5ff79fa71a0c: Deleting consensus metadata
I20260925 10:36:53.651288 17524 catalog_manager.cc:4841] TS b13095b1bd71450c809d867430586fa4 (127.0.97.2:37853): tablet c94d3e4a065d4f11aecf1416d38e6ef0 (table test-table [id=cbdb0488033a461d9800114550bb8042]) successfully deleted
I20260925 10:36:53.652094 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.586663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.0.97.1:32933)
0925 10:36:53.586919 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:53.586928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:53.612554 (+ 25626us) server_negotiation.cc:407] Connection header received
0925 10:36:53.614067 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:53.614075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:53.614217 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:53.614359 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:53.618954 (+ 4595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.620686 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.630449 (+ 9763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.631455 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.631595 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.643565 (+ 11970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:53.643584 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:53.643601 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:53.643664 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:53.647740 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:53.648711 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:53.648719 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:53.648722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:53.648907 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:53.649464 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:53.649471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:53.649475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:53.649955 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:53.650288 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:53.651482 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0925 10:36:53.651716 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:36:53.652103 17525 catalog_manager.cc:4841] TS fc5a2457b3bb47a29dfc5ff79fa71a0c (127.0.97.3:46047): tablet c94d3e4a065d4f11aecf1416d38e6ef0 (table test-table [id=cbdb0488033a461d9800114550bb8042]) successfully deleted
I20260925 10:36:53.652873 17722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:36:53.758720 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 17599
I20260925 10:36:53.777491 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 17733
W20260925 10:36:53.777450 17524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:32933 (error 108)
I20260925 10:36:53.779381 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.778862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32933 (local address 127.0.0.1:47192)
0925 10:36:53.779062 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:53.779194 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 10:36:53.779644 17524 catalog_manager.cc:4584] TS 30c988d404314a6daae06214cd6b2a60 (127.0.97.1:32933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c94d3e4a065d4f11aecf1416d38e6ef0: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32933: connect: Connection refused (error 111)
Sep 25 10:36:53 dist-test-slave-dxdd krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:36:53.791695 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.778255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52546 (local address 127.0.97.2:37853)
0925 10:36:53.778528 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:53.778535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:53.778631 (+ 96us) server_negotiation.cc:407] Connection header received
0925 10:36:53.778810 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:53.778816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:53.778905 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:53.779022 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:53.780386 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.781084 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.782259 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.782702 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.782792 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.786651 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:53.786666 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:53.786680 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:53.786708 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:53.789199 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:53.789814 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:53.789821 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:53.789823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:53.789934 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:53.790248 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:53.790251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:53.790253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:53.790889 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:53.791097 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:53.791310 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 10:36:53.791454 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260925 10:36:53.792330 17856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:36:53.833103 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.832586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32933 (local address 127.0.0.1:47200)
0925 10:36:53.832815 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:53.832925 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 10:36:53.834157 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.833752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32933 (local address 127.0.0.1:47216)
0925 10:36:53.833901 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:53.833990 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 10:36:53.881834 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 17733
I20260925 10:36:53.894788 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.894361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32933 (local address 127.0.0.1:47220)
0925 10:36:53.894556 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:53.894645 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 10:36:53.895867 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.895570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32933 (local address 127.0.0.1:47230)
0925 10:36:53.895695 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:53.895759 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 10:36:53.899765 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 17866
Sep 25 10:36:53 dist-test-slave-dxdd krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:36:53.913357 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.900433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44848 (local address 127.0.97.3:46047)
0925 10:36:53.900586 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:53.900590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:53.900608 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:36:53.900721 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:53.900725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:53.900789 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:53.900870 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:53.902102 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.902886 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.904080 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:53.904446 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:53.904531 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:53.908466 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:53.908481 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:53.908497 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:53.908533 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:53.911151 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:53.911764 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:53.911771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:53.911774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:53.911904 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:53.912197 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:53.912200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:53.912202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:53.912501 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:53.912712 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:53.912965 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 10:36:53.913116 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 10:36:53.913962 17989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:36:53.976701 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.976211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32933 (local address 127.0.0.1:47246)
0925 10:36:53.976426 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:53.976533 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 10:36:53.977758 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:53.977326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32933 (local address 127.0.0.1:47258)
0925 10:36:53.977527 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:53.977607 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 10:36:54.001705 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 17866
I20260925 10:36:54.018071 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 17506
I20260925 10:36:54.019228 17568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:36:54.126770 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 17506
2026-09-25T10:36:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4149 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T10:36:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:36:54Z Disabled control of system clock
I20260925 10:36:54.199406 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35581
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:36:54.516232 18048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:54.516520 18048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:54.516610 18048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:54.523736 18048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:36:54.523838 18048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:54.523875 18048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:54.523936 18048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:36:54.523972 18048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:36:54.532877 18048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:54.534046 18048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:54.535679 18048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:54.547360 18056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:54.547390 18054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:54.547844 18053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:54.548357 18048 server_base.cc:1034] running on GCE node
I20260925 10:36:54.549145 18048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:54.550496 18048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:54.551842 18048 hybrid_clock.cc:648] HybridClock initialized: now 1790332614551806 us; error 40 us; skew 500 ppm
I20260925 10:36:54.552201 18048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:36:54.554400 18048 webserver.cc:460] Webserver started at http://127.0.97.62:44813/ using document root <none> and password file <none>
I20260925 10:36:54.555037 18048 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:54.555166 18048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:54.555470 18048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:36:54.558365 18048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "3c182a5e6d2f4330a951123d8363db05"
format_stamp: "Formatted at 2026-09-25 10:36:54 on dist-test-slave-dxdd"
server_key: "8b4a04509f75e29732ab61eb91022bba"
server_key_iv: "0135e3772a2853caa3926fbed5797a57"
server_key_version: "encryptionkey@0"
I20260925 10:36:54.559135 18048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "3c182a5e6d2f4330a951123d8363db05"
format_stamp: "Formatted at 2026-09-25 10:36:54 on dist-test-slave-dxdd"
server_key: "8b4a04509f75e29732ab61eb91022bba"
server_key_iv: "0135e3772a2853caa3926fbed5797a57"
server_key_version: "encryptionkey@0"
I20260925 10:36:54.564196 18048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:36:54.567329 18062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:54.568642 18048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:36:54.568812 18048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "3c182a5e6d2f4330a951123d8363db05"
format_stamp: "Formatted at 2026-09-25 10:36:54 on dist-test-slave-dxdd"
server_key: "8b4a04509f75e29732ab61eb91022bba"
server_key_iv: "0135e3772a2853caa3926fbed5797a57"
server_key_version: "encryptionkey@0"
I20260925 10:36:54.569005 18048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:54.587028 18048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:54.587772 18048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:54.588061 18048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:54.605088 18048 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35581
I20260925 10:36:54.605108 18113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35581 every 8 connection(s)
I20260925 10:36:54.606400 18048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:36:54.608788 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 18048
I20260925 10:36:54.608949 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:36:54.609392 388 external_mini_cluster.cc:1428] Setting key a1602e7ab55fc8bd18814bc1bb280190
I20260925 10:36:54.610248 18114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:36:54.623373 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:54.611368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40500 (local address 127.0.97.62:35581)
0925 10:36:54.612324 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:54.612334 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:54.612372 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:36:54.614741 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:54.614761 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:54.615186 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:54.615485 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:54.616066 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:54.617274 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:54.619170 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:54.619786 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:54.619917 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:54.621240 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:54.621247 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:54.621263 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:36:54.621762 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:54.621832 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:36:54.621861 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:54.621943 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:54.622163 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 10:36:54.622537 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":369,"threads_started":1}
I20260925 10:36:54.623607 18114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05: Bootstrap starting.
I20260925 10:36:54.626873 18114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05: Neither blocks nor log segments found. Creating new log.
I20260925 10:36:54.627709 18114 log.cc:824] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05: Log is configured to *not* fsync() on all Append() calls
I20260925 10:36:54.630366 18114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05: No bootstrap required, opened a new log
I20260925 10:36:54.637928 18114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c182a5e6d2f4330a951123d8363db05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35581 } }
I20260925 10:36:54.638350 18114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:36:54.638450 18114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c182a5e6d2f4330a951123d8363db05, State: Initialized, Role: FOLLOWER
I20260925 10:36:54.639106 18114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c182a5e6d2f4330a951123d8363db05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35581 } }
I20260925 10:36:54.639299 18114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:36:54.639394 18114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:36:54.639511 18114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:36:54.641860 18114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c182a5e6d2f4330a951123d8363db05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35581 } }
I20260925 10:36:54.642268 18114 leader_election.cc:302] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c182a5e6d2f4330a951123d8363db05; no voters:
I20260925 10:36:54.642843 18114 leader_election.cc:288] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:36:54.642968 18119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:36:54.643640 18119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [term 1 LEADER]: Becoming Leader. State: Replica: 3c182a5e6d2f4330a951123d8363db05, State: Running, Role: LEADER
I20260925 10:36:54.644079 18119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c182a5e6d2f4330a951123d8363db05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35581 } }
I20260925 10:36:54.644745 18114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:36:54.646919 18121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c182a5e6d2f4330a951123d8363db05. Latest consensus state: current_term: 1 leader_uuid: "3c182a5e6d2f4330a951123d8363db05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c182a5e6d2f4330a951123d8363db05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35581 } } }
I20260925 10:36:54.647310 18121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [sys.catalog]: This master's current role is: LEADER
I20260925 10:36:54.647740 18120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c182a5e6d2f4330a951123d8363db05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c182a5e6d2f4330a951123d8363db05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35581 } } }
I20260925 10:36:54.648458 18120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05 [sys.catalog]: This master's current role is: LEADER
I20260925 10:36:54.650418 18126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:36:54.655508 18126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:36:54.663920 18126 catalog_manager.cc:1347] Generated new cluster ID: 465832492c33451b8e7635fa024c3f28
I20260925 10:36:54.664062 18126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:36:54.685819 18126 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:36:54.687095 18126 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:36:54.693073 18126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05: Generated new TSK 0
I20260925 10:36:54.693996 18126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:36:54.708913 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35581
--builtin_ntp_servers=127.0.97.20:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:36:55.029436 18138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:55.029733 18138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:55.029904 18138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:55.037043 18138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:55.037173 18138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:55.037369 18138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:36:55.047650 18138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:55.049304 18138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:55.051973 18138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:55.065372 18146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:55.065462 18143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:55.065397 18144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:55.066530 18138 server_base.cc:1034] running on GCE node
I20260925 10:36:55.067153 18138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:55.068190 18138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:55.069397 18138 hybrid_clock.cc:648] HybridClock initialized: now 1790332615069355 us; error 38 us; skew 500 ppm
I20260925 10:36:55.069686 18138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:36:55.076705 18138 webserver.cc:460] Webserver started at http://127.0.97.1:38535/ using document root <none> and password file <none>
I20260925 10:36:55.077373 18138 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:55.077484 18138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:55.077800 18138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:36:55.080596 18138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "a5bc20f72b6146ed824b96f6208555d9"
format_stamp: "Formatted at 2026-09-25 10:36:55 on dist-test-slave-dxdd"
server_key: "89c85b0f1447fc7e5a53dd9eb837614d"
server_key_iv: "9bf8d0c6bb01d479cf2d8bb31282819d"
server_key_version: "encryptionkey@0"
I20260925 10:36:55.081346 18138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a5bc20f72b6146ed824b96f6208555d9"
format_stamp: "Formatted at 2026-09-25 10:36:55 on dist-test-slave-dxdd"
server_key: "89c85b0f1447fc7e5a53dd9eb837614d"
server_key_iv: "9bf8d0c6bb01d479cf2d8bb31282819d"
server_key_version: "encryptionkey@0"
I20260925 10:36:55.086042 18138 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260925 10:36:55.089275 18152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:55.090528 18138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:36:55.090674 18138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "a5bc20f72b6146ed824b96f6208555d9"
format_stamp: "Formatted at 2026-09-25 10:36:55 on dist-test-slave-dxdd"
server_key: "89c85b0f1447fc7e5a53dd9eb837614d"
server_key_iv: "9bf8d0c6bb01d479cf2d8bb31282819d"
server_key_version: "encryptionkey@0"
I20260925 10:36:55.090874 18138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:55.112999 18138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:55.113741 18138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:55.114015 18138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:55.114940 18138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:36:55.116302 18138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:36:55.116408 18138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:55.116528 18138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:36:55.116622 18138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:55.148653 18138 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36461
I20260925 10:36:55.148715 18264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36461 every 8 connection(s)
I20260925 10:36:55.149932 18138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:36:55.154227 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 18138
I20260925 10:36:55.154426 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:36:55.154870 388 external_mini_cluster.cc:1428] Setting key a3e271253e6dd6547079f7b4921d4b67
I20260925 10:36:55.157697 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35581
--builtin_ntp_servers=127.0.97.20:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:36:55.163461 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:55.152482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52175 (local address 127.0.97.62:35581)
0925 10:36:55.152904 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:55.152911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:55.154085 (+ 1174us) server_negotiation.cc:407] Connection header received
0925 10:36:55.156253 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:55.156258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:55.156359 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:55.156487 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:55.157645 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:55.158846 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:55.160345 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:55.161058 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:55.161194 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:55.162524 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:55.162533 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:55.162541 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:36:55.162567 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:55.162618 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:36:55.162636 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:55.162734 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:55.162928 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 10:36:55.163040 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":202,"threads_started":1}
I20260925 10:36:55.164192 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:55.153101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35581 (local address 127.0.97.1:52175)
0925 10:36:55.153931 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:55.153967 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:55.155955 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:55.156777 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:55.156788 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:55.156925 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:36:55.157476 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:55.157492 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:55.158998 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:55.159001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:55.160196 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:55.160207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:55.161249 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:55.161253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:55.161447 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:55.162155 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:36:55.162157 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:36:55.162195 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:36:55.162287 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:36:55.162735 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:36:55.162743 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:36:55.162748 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:36:55.162883 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 10:36:55.163155 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":252,"threads_started":1}
I20260925 10:36:55.165741 18265 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35581
I20260925 10:36:55.166076 18265 heartbeater.cc:464] Registering TS with master...
I20260925 10:36:55.166994 18265 heartbeater.cc:511] Master 127.0.97.62:35581 requested a full tablet report, sending...
I20260925 10:36:55.169020 18079 ts_manager.cc:194] Registered new tserver with Master: a5bc20f72b6146ed824b96f6208555d9 (127.0.97.1:36461)
I20260925 10:36:55.170575 18079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:52175
W20260925 10:36:55.471971 18269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:55.472218 18269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:55.472369 18269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:55.479426 18269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:55.479512 18269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:55.479612 18269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:36:55.488219 18269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:55.489320 18269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:55.490935 18269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:55.503785 18274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:55.503739 18277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:55.503818 18275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:55.504760 18269 server_base.cc:1034] running on GCE node
I20260925 10:36:55.505352 18269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:55.506254 18269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:55.507495 18269 hybrid_clock.cc:648] HybridClock initialized: now 1790332615507457 us; error 36 us; skew 500 ppm
I20260925 10:36:55.507805 18269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:36:55.509894 18269 webserver.cc:460] Webserver started at http://127.0.97.2:33301/ using document root <none> and password file <none>
I20260925 10:36:55.510635 18269 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:55.510766 18269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:55.511077 18269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:36:55.513958 18269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "4e52b05b7b0c40f1ae1c02cd05127453"
format_stamp: "Formatted at 2026-09-25 10:36:55 on dist-test-slave-dxdd"
server_key: "2d6eb6c9969409baac67bb71c655a640"
server_key_iv: "da34e65f480ab79cca3bd823c615ade8"
server_key_version: "encryptionkey@0"
I20260925 10:36:55.514750 18269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4e52b05b7b0c40f1ae1c02cd05127453"
format_stamp: "Formatted at 2026-09-25 10:36:55 on dist-test-slave-dxdd"
server_key: "2d6eb6c9969409baac67bb71c655a640"
server_key_iv: "da34e65f480ab79cca3bd823c615ade8"
server_key_version: "encryptionkey@0"
I20260925 10:36:55.519424 18269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:36:55.522754 18283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:55.523967 18269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:36:55.524128 18269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "4e52b05b7b0c40f1ae1c02cd05127453"
format_stamp: "Formatted at 2026-09-25 10:36:55 on dist-test-slave-dxdd"
server_key: "2d6eb6c9969409baac67bb71c655a640"
server_key_iv: "da34e65f480ab79cca3bd823c615ade8"
server_key_version: "encryptionkey@0"
I20260925 10:36:55.524330 18269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:55.537925 18269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:55.538669 18269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:55.538938 18269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:55.539798 18269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:36:55.541188 18269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:36:55.541307 18269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:55.541435 18269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:36:55.541522 18269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:55.573733 18269 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41227
I20260925 10:36:55.573778 18395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41227 every 8 connection(s)
I20260925 10:36:55.575021 18269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:36:55.582954 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 18269
I20260925 10:36:55.583138 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:36:55.583709 388 external_mini_cluster.cc:1428] Setting key 07449ce3bcbe2390864d915bec7f8c6a
I20260925 10:36:55.586843 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35581
--builtin_ntp_servers=127.0.97.20:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:36:55.587772 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:55.577382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44571 (local address 127.0.97.62:35581)
0925 10:36:55.577565 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:55.577569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:55.578977 (+ 1408us) server_negotiation.cc:407] Connection header received
0925 10:36:55.580598 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:55.580603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:55.580666 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:55.580752 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:55.581728 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:55.582496 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:55.584689 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:55.585265 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:55.585378 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:55.586918 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:55.586923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:55.586927 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:36:55.586969 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:55.587013 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:36:55.587035 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:55.587130 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:55.587354 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 10:36:55.587448 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":71}
I20260925 10:36:55.588790 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:55.577903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35581 (local address 127.0.97.2:44571)
0925 10:36:55.578818 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:55.578858 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:55.580361 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:55.580953 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:55.580960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:55.581040 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:36:55.581585 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:55.581599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:55.582650 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:55.582656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:55.584535 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:55.584550 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:55.585485 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:55.585495 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:55.585756 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:55.586569 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:36:55.586574 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:36:55.586628 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:36:55.586724 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:36:55.587139 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:36:55.587152 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:36:55.587167 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:36:55.587344 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 10:36:55.587724 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":243,"threads_started":1}
I20260925 10:36:55.590112 18396 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35581
I20260925 10:36:55.590440 18396 heartbeater.cc:464] Registering TS with master...
I20260925 10:36:55.591128 18396 heartbeater.cc:511] Master 127.0.97.62:35581 requested a full tablet report, sending...
I20260925 10:36:55.592625 18079 ts_manager.cc:194] Registered new tserver with Master: 4e52b05b7b0c40f1ae1c02cd05127453 (127.0.97.2:41227)
I20260925 10:36:55.593828 18079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:44571
W20260925 10:36:55.905741 18399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:55.906011 18399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:55.906095 18399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:55.913403 18399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:55.913506 18399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:55.913609 18399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:36:55.922817 18399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:55.924021 18399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:55.925678 18399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:55.938665 18404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:55.938717 18405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:55.938757 18407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:55.939419 18399 server_base.cc:1034] running on GCE node
I20260925 10:36:55.939987 18399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:55.940889 18399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:55.942112 18399 hybrid_clock.cc:648] HybridClock initialized: now 1790332615942076 us; error 60 us; skew 500 ppm
I20260925 10:36:55.942507 18399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:36:55.944777 18399 webserver.cc:460] Webserver started at http://127.0.97.3:33681/ using document root <none> and password file <none>
I20260925 10:36:55.945467 18399 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:55.945602 18399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:55.945916 18399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:36:55.948683 18399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "33fd2800ca114fbfae1f57199bf12585"
format_stamp: "Formatted at 2026-09-25 10:36:55 on dist-test-slave-dxdd"
server_key: "278d0eb01614f76a49682659a06233fe"
server_key_iv: "69e289079c399c57994da01f32874c13"
server_key_version: "encryptionkey@0"
I20260925 10:36:55.949465 18399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "33fd2800ca114fbfae1f57199bf12585"
format_stamp: "Formatted at 2026-09-25 10:36:55 on dist-test-slave-dxdd"
server_key: "278d0eb01614f76a49682659a06233fe"
server_key_iv: "69e289079c399c57994da01f32874c13"
server_key_version: "encryptionkey@0"
I20260925 10:36:55.954454 18399 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 10:36:55.957733 18413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:55.958981 18399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:36:55.959151 18399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "33fd2800ca114fbfae1f57199bf12585"
format_stamp: "Formatted at 2026-09-25 10:36:55 on dist-test-slave-dxdd"
server_key: "278d0eb01614f76a49682659a06233fe"
server_key_iv: "69e289079c399c57994da01f32874c13"
server_key_version: "encryptionkey@0"
I20260925 10:36:55.959372 18399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:55.987468 18399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:55.988366 18399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:55.988658 18399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:55.989636 18399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:36:55.991133 18399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:36:55.991259 18399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:55.991395 18399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:36:55.991482 18399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:56.023881 18399 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35157
I20260925 10:36:56.023972 18525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35157 every 8 connection(s)
I20260925 10:36:56.025178 18399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:36:56.030727 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 18399
I20260925 10:36:56.030898 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:36:56.031363 388 external_mini_cluster.cc:1428] Setting key 0da7249a3c3edd4063420c738a4819d4
I20260925 10:36:56.037413 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.027646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53323 (local address 127.0.97.62:35581)
0925 10:36:56.027825 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:56.027829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:56.029252 (+ 1423us) server_negotiation.cc:407] Connection header received
0925 10:36:56.031119 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:56.031124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:56.031215 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:56.031330 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:56.032396 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.033202 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.034788 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.035210 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.035358 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.036610 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:56.036615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:56.036618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:36:56.036637 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:56.036669 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:36:56.036680 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:56.036775 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:56.037004 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 10:36:56.037097 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:36:56.038445 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.028196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35581 (local address 127.0.97.3:53323)
0925 10:36:56.029113 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:56.029147 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:56.030774 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:56.031568 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:56.031576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:56.031663 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:36:56.032219 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:56.032234 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.033353 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.033359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:56.034649 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:56.034661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.035429 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.035436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:56.035667 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.036305 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:36:56.036308 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:36:56.036348 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:36:56.036421 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:36:56.036772 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:36:56.036783 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:36:56.036793 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 10:36:56.036981 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 10:36:56.037356 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":267,"threads_started":1}
I20260925 10:36:56.039642 18526 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35581
I20260925 10:36:56.039889 18526 heartbeater.cc:464] Registering TS with master...
I20260925 10:36:56.040518 18526 heartbeater.cc:511] Master 127.0.97.62:35581 requested a full tablet report, sending...
I20260925 10:36:56.041751 18079 ts_manager.cc:194] Registered new tserver with Master: 33fd2800ca114fbfae1f57199bf12585 (127.0.97.3:35157)
I20260925 10:36:56.042546 18079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:53323
I20260925 10:36:56.046111 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:36:56.055840 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.049865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40502 (local address 127.0.97.62:35581)
0925 10:36:56.050083 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:56.050092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:56.050455 (+ 363us) server_negotiation.cc:407] Connection header received
0925 10:36:56.050646 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:56.050651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:56.050725 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:56.050834 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:56.051366 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.052040 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.053448 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.053899 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.054036 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.055179 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:56.055184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:56.055188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:36:56.055204 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:56.055235 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:36:56.055246 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:56.055342 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:56.055460 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 10:36:56.055549 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:36:56.059466 18079 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:36:56.066206 18079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:36:56.092187 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.077809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36461 (local address 127.0.0.1:54610)
0925 10:36:56.080938 (+ 3129us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:56.080956 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:56.081079 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:56.082144 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:56.082152 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:56.082185 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:36:56.082647 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:56.082663 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.085069 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.085077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:56.088140 (+ 3063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:56.088160 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.091019 (+ 2859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.091028 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:56.091302 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.091365 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 10:36:56.091457 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 10:36:56.091642 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2965,"mutex_wait_us":7,"thread_start_us":2842,"threads_started":1}
I20260925 10:36:56.092533 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.078603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.0.97.2:41227)
0925 10:36:56.080408 (+ 1805us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:56.080418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:56.080449 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:36:56.080552 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:56.080562 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:56.080850 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:56.081054 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:36:56.082051 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.084243 (+ 2192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.087940 (+ 3697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.089592 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.089755 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.090120 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:56.091801 (+ 1681us) server_negotiation.cc:299] Negotiation successful
0925 10:36:56.092035 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1604,"thread_start_us":1470,"threads_started":1}
I20260925 10:36:56.093222 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.077880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.0.97.1:36461)
0925 10:36:56.080541 (+ 2661us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:56.080548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:56.081206 (+ 658us) server_negotiation.cc:407] Connection header received
0925 10:36:56.081326 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:56.081334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:56.081585 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:56.081811 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:36:56.082861 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.084862 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.088425 (+ 3563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.090392 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.090553 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.090854 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:56.092517 (+ 1663us) server_negotiation.cc:299] Negotiation successful
0925 10:36:56.092762 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2514,"thread_start_us":2422,"threads_started":1}
I20260925 10:36:56.093461 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.077805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41227 (local address 127.0.0.1:56862)
0925 10:36:56.079792 (+ 1987us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:56.079838 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:56.080011 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:56.081369 (+ 1358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:56.081380 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:56.081411 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:36:56.081797 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:56.081811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.084414 (+ 2603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.084421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:56.087648 (+ 3227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:56.087669 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.090611 (+ 2942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.090620 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:56.090938 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.091015 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 10:36:56.091180 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 10:36:56.091458 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1749,"mutex_wait_us":131,"thread_start_us":1584,"threads_started":1}
I20260925 10:36:56.097750 18331 tablet_service.cc:1467] Processing CreateTablet for tablet 4838e579469d4b628f30f014c9f2b109 (DEFAULT_TABLE table=test-table [id=968026875ac14b018828f2ed39631347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:36:56.098158 18200 tablet_service.cc:1467] Processing CreateTablet for tablet 4838e579469d4b628f30f014c9f2b109 (DEFAULT_TABLE table=test-table [id=968026875ac14b018828f2ed39631347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:36:56.099224 18331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4838e579469d4b628f30f014c9f2b109. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:36:56.100081 18200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4838e579469d4b628f30f014c9f2b109. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:36:56.101212 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.081764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35157 (local address 127.0.0.1:42108)
0925 10:36:56.093830 (+ 12066us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:56.093854 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:56.093976 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:56.096126 (+ 2150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:56.096131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:56.096155 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:36:56.096321 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:56.096332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.097583 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.097591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:56.098960 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:56.098973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.100506 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.100511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:56.100683 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.100724 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 10:36:56.100810 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 10:36:56.100936 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11914,"thread_start_us":12794,"threads_started":1}
I20260925 10:36:56.101356 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.081369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.0.97.3:35157)
0925 10:36:56.095669 (+ 14300us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:56.095675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:56.095701 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:36:56.095771 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:56.095777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:56.095947 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:56.096147 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:36:56.096454 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.097445 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.099142 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.100350 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.100496 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.100747 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:56.100853 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 10:36:56.100998 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14147,"thread_start_us":13957,"threads_started":1}
I20260925 10:36:56.104882 18461 tablet_service.cc:1467] Processing CreateTablet for tablet 4838e579469d4b628f30f014c9f2b109 (DEFAULT_TABLE table=test-table [id=968026875ac14b018828f2ed39631347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:36:56.105955 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4838e579469d4b628f30f014c9f2b109. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:36:56.113628 18544 tablet_bootstrap.cc:492] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453: Bootstrap starting.
I20260925 10:36:56.115769 18545 tablet_bootstrap.cc:492] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9: Bootstrap starting.
I20260925 10:36:56.117570 18546 tablet_bootstrap.cc:492] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585: Bootstrap starting.
I20260925 10:36:56.118166 18544 tablet_bootstrap.cc:654] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453: Neither blocks nor log segments found. Creating new log.
I20260925 10:36:56.119186 18544 log.cc:824] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453: Log is configured to *not* fsync() on all Append() calls
I20260925 10:36:56.119220 18545 tablet_bootstrap.cc:654] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9: Neither blocks nor log segments found. Creating new log.
I20260925 10:36:56.120409 18545 log.cc:824] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:36:56.120887 18546 tablet_bootstrap.cc:654] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585: Neither blocks nor log segments found. Creating new log.
I20260925 10:36:56.121903 18546 log.cc:824] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585: Log is configured to *not* fsync() on all Append() calls
I20260925 10:36:56.122042 18544 tablet_bootstrap.cc:492] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453: No bootstrap required, opened a new log
I20260925 10:36:56.122372 18544 ts_tablet_manager.cc:1397] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 10:36:56.123366 18545 tablet_bootstrap.cc:492] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9: No bootstrap required, opened a new log
I20260925 10:36:56.123611 18545 ts_tablet_manager.cc:1397] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 10:36:56.124910 18546 tablet_bootstrap.cc:492] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585: No bootstrap required, opened a new log
I20260925 10:36:56.126434 18546 ts_tablet_manager.cc:1397] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260925 10:36:56.129890 18544 raft_consensus.cc:348] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bc20f72b6146ed824b96f6208555d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36461 } } peers { permanent_uuid: "4e52b05b7b0c40f1ae1c02cd05127453" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41227 } } peers { permanent_uuid: "33fd2800ca114fbfae1f57199bf12585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35157 } }
I20260925 10:36:56.130235 18544 raft_consensus.cc:374] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:36:56.130381 18544 raft_consensus.cc:729] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e52b05b7b0c40f1ae1c02cd05127453, State: Initialized, Role: FOLLOWER
I20260925 10:36:56.130944 18544 consensus_queue.cc:260] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bc20f72b6146ed824b96f6208555d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36461 } } peers { permanent_uuid: "4e52b05b7b0c40f1ae1c02cd05127453" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41227 } } peers { permanent_uuid: "33fd2800ca114fbfae1f57199bf12585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35157 } }
I20260925 10:36:56.131980 18396 heartbeater.cc:503] Master 127.0.97.62:35581 was elected leader, sending a full tablet report...
I20260925 10:36:56.132596 18544 ts_tablet_manager.cc:1428] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453: Time spent starting tablet: real 0.010s user 0.004s sys 0.005s
I20260925 10:36:56.133934 18545 raft_consensus.cc:348] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bc20f72b6146ed824b96f6208555d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36461 } } peers { permanent_uuid: "4e52b05b7b0c40f1ae1c02cd05127453" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41227 } } peers { permanent_uuid: "33fd2800ca114fbfae1f57199bf12585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35157 } }
I20260925 10:36:56.134371 18545 raft_consensus.cc:374] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:36:56.134473 18545 raft_consensus.cc:729] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5bc20f72b6146ed824b96f6208555d9, State: Initialized, Role: FOLLOWER
I20260925 10:36:56.135017 18545 consensus_queue.cc:260] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bc20f72b6146ed824b96f6208555d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36461 } } peers { permanent_uuid: "4e52b05b7b0c40f1ae1c02cd05127453" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41227 } } peers { permanent_uuid: "33fd2800ca114fbfae1f57199bf12585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35157 } }
I20260925 10:36:56.136133 18265 heartbeater.cc:503] Master 127.0.97.62:35581 was elected leader, sending a full tablet report...
I20260925 10:36:56.136725 18545 ts_tablet_manager.cc:1428] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 10:36:56.137468 18546 raft_consensus.cc:348] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bc20f72b6146ed824b96f6208555d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36461 } } peers { permanent_uuid: "4e52b05b7b0c40f1ae1c02cd05127453" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41227 } } peers { permanent_uuid: "33fd2800ca114fbfae1f57199bf12585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35157 } }
I20260925 10:36:56.137864 18546 raft_consensus.cc:374] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:36:56.137969 18546 raft_consensus.cc:729] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33fd2800ca114fbfae1f57199bf12585, State: Initialized, Role: FOLLOWER
I20260925 10:36:56.138962 18546 consensus_queue.cc:260] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bc20f72b6146ed824b96f6208555d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36461 } } peers { permanent_uuid: "4e52b05b7b0c40f1ae1c02cd05127453" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41227 } } peers { permanent_uuid: "33fd2800ca114fbfae1f57199bf12585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35157 } }
I20260925 10:36:56.140039 18526 heartbeater.cc:503] Master 127.0.97.62:35581 was elected leader, sending a full tablet report...
I20260925 10:36:56.140609 18546 ts_tablet_manager.cc:1428] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260925 10:36:56.152161 18266 tablet.cc:2369] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:36:56.213567 18552 raft_consensus.cc:482] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:36:56.213912 18552 raft_consensus.cc:504] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bc20f72b6146ed824b96f6208555d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36461 } } peers { permanent_uuid: "4e52b05b7b0c40f1ae1c02cd05127453" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41227 } } peers { permanent_uuid: "33fd2800ca114fbfae1f57199bf12585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35157 } }
I20260925 10:36:56.215617 18552 leader_election.cc:288] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5bc20f72b6146ed824b96f6208555d9 (127.0.97.1:36461), 4e52b05b7b0c40f1ae1c02cd05127453 (127.0.97.2:41227)
I20260925 10:36:56.221283 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.216084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35417 (local address 127.0.97.2:41227)
0925 10:36:56.216266 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:56.216272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:56.216445 (+ 173us) server_negotiation.cc:407] Connection header received
0925 10:36:56.216649 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:56.216653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:56.216711 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:56.216827 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:36:56.217258 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.217964 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.219655 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.220409 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.220547 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.220643 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:56.220929 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 10:36:56.221050 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:36:56.221292 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.216074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41227 (local address 127.0.97.3:35417)
0925 10:36:56.216338 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:56.216358 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:56.216505 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:56.216857 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:56.216862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:56.216908 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:36:56.217116 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:56.217124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.218139 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.218143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:56.219509 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:56.219520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.220559 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.220565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:56.220745 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.220767 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 10:36:56.220881 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 10:36:56.221019 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":1}
I20260925 10:36:56.222086 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.216034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38225 (local address 127.0.97.1:36461)
0925 10:36:56.216223 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:56.216227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:56.217330 (+ 1103us) server_negotiation.cc:407] Connection header received
0925 10:36:56.217520 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:56.217523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:56.217589 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:56.217688 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:36:56.218161 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.218883 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.220533 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.221243 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.221387 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.221469 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:56.221738 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 10:36:56.221855 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:36:56.222182 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.216074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36461 (local address 127.0.97.3:38225)
0925 10:36:56.217215 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:56.217228 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:56.217359 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:56.217700 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:56.217705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:56.217725 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:36:56.218012 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:56.218019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.219025 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.219030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:56.220359 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:56.220369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.221402 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.221407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:56.221581 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.221602 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 10:36:56.221699 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 10:36:56.221822 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":229,"thread_start_us":519,"threads_started":1}
I20260925 10:36:56.222656 18351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4838e579469d4b628f30f014c9f2b109" candidate_uuid: "33fd2800ca114fbfae1f57199bf12585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e52b05b7b0c40f1ae1c02cd05127453" is_pre_election: true
I20260925 10:36:56.223152 18351 raft_consensus.cc:2454] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33fd2800ca114fbfae1f57199bf12585 in term 0.
I20260925 10:36:56.223145 18220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4838e579469d4b628f30f014c9f2b109" candidate_uuid: "33fd2800ca114fbfae1f57199bf12585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5bc20f72b6146ed824b96f6208555d9" is_pre_election: true
I20260925 10:36:56.223627 18220 raft_consensus.cc:2454] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33fd2800ca114fbfae1f57199bf12585 in term 0.
I20260925 10:36:56.223892 18417 leader_election.cc:302] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33fd2800ca114fbfae1f57199bf12585, 4e52b05b7b0c40f1ae1c02cd05127453; no voters:
I20260925 10:36:56.224293 18552 raft_consensus.cc:2781] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:36:56.224431 18552 raft_consensus.cc:482] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:36:56.224529 18552 raft_consensus.cc:3037] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:36:56.226698 18552 raft_consensus.cc:504] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bc20f72b6146ed824b96f6208555d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36461 } } peers { permanent_uuid: "4e52b05b7b0c40f1ae1c02cd05127453" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41227 } } peers { permanent_uuid: "33fd2800ca114fbfae1f57199bf12585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35157 } }
I20260925 10:36:56.227231 18552 leader_election.cc:288] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [CANDIDATE]: Term 1 election: Requested vote from peers a5bc20f72b6146ed824b96f6208555d9 (127.0.97.1:36461), 4e52b05b7b0c40f1ae1c02cd05127453 (127.0.97.2:41227)
I20260925 10:36:56.227610 18220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4838e579469d4b628f30f014c9f2b109" candidate_uuid: "33fd2800ca114fbfae1f57199bf12585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5bc20f72b6146ed824b96f6208555d9"
I20260925 10:36:56.227844 18220 raft_consensus.cc:3037] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:36:56.227888 18351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4838e579469d4b628f30f014c9f2b109" candidate_uuid: "33fd2800ca114fbfae1f57199bf12585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e52b05b7b0c40f1ae1c02cd05127453"
I20260925 10:36:56.228111 18351 raft_consensus.cc:3037] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:36:56.231087 18220 raft_consensus.cc:2454] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33fd2800ca114fbfae1f57199bf12585 in term 1.
I20260925 10:36:56.231252 18351 raft_consensus.cc:2454] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33fd2800ca114fbfae1f57199bf12585 in term 1.
I20260925 10:36:56.231493 18415 leader_election.cc:302] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33fd2800ca114fbfae1f57199bf12585, a5bc20f72b6146ed824b96f6208555d9; no voters:
I20260925 10:36:56.231819 18552 raft_consensus.cc:2781] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:36:56.232230 18552 raft_consensus.cc:686] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [term 1 LEADER]: Becoming Leader. State: Replica: 33fd2800ca114fbfae1f57199bf12585, State: Running, Role: LEADER
I20260925 10:36:56.232657 18552 consensus_queue.cc:237] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bc20f72b6146ed824b96f6208555d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36461 } } peers { permanent_uuid: "4e52b05b7b0c40f1ae1c02cd05127453" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41227 } } peers { permanent_uuid: "33fd2800ca114fbfae1f57199bf12585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35157 } }
I20260925 10:36:56.236678 18078 catalog_manager.cc:5495] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33fd2800ca114fbfae1f57199bf12585 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "33fd2800ca114fbfae1f57199bf12585" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bc20f72b6146ed824b96f6208555d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e52b05b7b0c40f1ae1c02cd05127453" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33fd2800ca114fbfae1f57199bf12585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35157 } health_report { overall_health: HEALTHY } } }
W20260925 10:36:56.276916 18527 tablet.cc:2369] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:36:56.278023 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.272960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42116 (local address 127.0.97.3:35157)
0925 10:36:56.273140 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:56.273148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:56.273292 (+ 144us) server_negotiation.cc:407] Connection header received
0925 10:36:56.273483 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:56.273487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:56.273560 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:56.273679 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:36:56.273953 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.274697 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.275986 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.276370 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.276459 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.276906 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:36:56.277582 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:36:56.277719 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:56.277794 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 10:36:56.277853 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:36:56.285676 18220 raft_consensus.cc:1261] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9 [term 1 FOLLOWER]: Refusing update from remote peer 33fd2800ca114fbfae1f57199bf12585: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:36:56.285679 18351 raft_consensus.cc:1261] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453 [term 1 FOLLOWER]: Refusing update from remote peer 33fd2800ca114fbfae1f57199bf12585: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:36:56.286589 18554 consensus_queue.cc:1035] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e52b05b7b0c40f1ae1c02cd05127453" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:36:56.286944 18552 consensus_queue.cc:1035] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5bc20f72b6146ed824b96f6208555d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:36:56.298894 18559 mvcc.cc:205] Tried to move back new op lower bound from 7333202396297359360 to 7333202396092809216. Current Snapshot: MvccSnapshot[applied={T|T < 7333202396297359360}]
I20260925 10:36:56.311233 18079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:36:56.311486 18079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:36:56.314878 18079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c182a5e6d2f4330a951123d8363db05: Sending DeleteTablet for 3 replicas of tablet 4838e579469d4b628f30f014c9f2b109
I20260925 10:36:56.315961 18331 tablet_service.cc:1514] Processing DeleteTablet for tablet 4838e579469d4b628f30f014c9f2b109 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:36:56 UTC) from {username='slave'} at 127.0.0.1:56862
I20260925 10:36:56.315969 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet 4838e579469d4b628f30f014c9f2b109 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:36:56 UTC) from {username='slave'} at 127.0.0.1:42108
I20260925 10:36:56.316051 18200 tablet_service.cc:1514] Processing DeleteTablet for tablet 4838e579469d4b628f30f014c9f2b109 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:36:56 UTC) from {username='slave'} at 127.0.0.1:54610
I20260925 10:36:56.318084 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 18138
I20260925 10:36:56.331347 18568 tablet_replica.cc:330] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9: stopping tablet replica
I20260925 10:36:56.331846 18568 raft_consensus.cc:2229] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:36:56.332384 18568 raft_consensus.cc:2258] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:36:56.334662 18568 ts_tablet_manager.cc:1905] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:36:56.342828 18568 ts_tablet_manager.cc:1918] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:36:56.342988 18568 log.cc:1196] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/4838e579469d4b628f30f014c9f2b109
I20260925 10:36:56.343566 18568 ts_tablet_manager.cc:1939] T 4838e579469d4b628f30f014c9f2b109 P a5bc20f72b6146ed824b96f6208555d9: Deleting consensus metadata
I20260925 10:36:56.345371 18064 catalog_manager.cc:4841] TS a5bc20f72b6146ed824b96f6208555d9 (127.0.97.1:36461): tablet 4838e579469d4b628f30f014c9f2b109 (table test-table [id=968026875ac14b018828f2ed39631347]) successfully deleted
I20260925 10:36:56.346503 18566 tablet_replica.cc:330] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453: stopping tablet replica
I20260925 10:36:56.346987 18566 raft_consensus.cc:2229] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:36:56.347611 18566 raft_consensus.cc:2258] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:36:56.349701 18566 ts_tablet_manager.cc:1905] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:36:56.352286 18567 tablet_replica.cc:330] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585: stopping tablet replica
I20260925 10:36:56.352751 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.319817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.0.97.1:36461)
0925 10:36:56.320028 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:56.320038 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:56.344200 (+ 24162us) server_negotiation.cc:407] Connection header received
0925 10:36:56.345058 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:56.345062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:56.345172 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:56.345241 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:56.346041 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.347175 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.349055 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.349646 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.349746 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.351221 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:56.351230 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:56.351246 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:36:56.351944 (+ 698us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:56.352000 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:36:56.352024 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:56.352160 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:56.352307 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 10:36:56.352421 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:36:56.353269 18567 raft_consensus.cc:2229] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:36:56.353461 18260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:36:56.353948 18567 raft_consensus.cc:2258] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:36:56.358202 18567 ts_tablet_manager.cc:1905] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:36:56.365615 18566 ts_tablet_manager.cc:1918] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:36:56.365841 18566 log.cc:1196] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/4838e579469d4b628f30f014c9f2b109
I20260925 10:36:56.365825 18567 ts_tablet_manager.cc:1918] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:36:56.365981 18567 log.cc:1196] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/4838e579469d4b628f30f014c9f2b109
I20260925 10:36:56.366506 18566 ts_tablet_manager.cc:1939] T 4838e579469d4b628f30f014c9f2b109 P 4e52b05b7b0c40f1ae1c02cd05127453: Deleting consensus metadata
I20260925 10:36:56.366600 18567 ts_tablet_manager.cc:1939] T 4838e579469d4b628f30f014c9f2b109 P 33fd2800ca114fbfae1f57199bf12585: Deleting consensus metadata
I20260925 10:36:56.368311 18066 catalog_manager.cc:4841] TS 4e52b05b7b0c40f1ae1c02cd05127453 (127.0.97.2:41227): tablet 4838e579469d4b628f30f014c9f2b109 (table test-table [id=968026875ac14b018828f2ed39631347]) successfully deleted
I20260925 10:36:56.368539 18064 catalog_manager.cc:4841] TS 33fd2800ca114fbfae1f57199bf12585 (127.0.97.3:35157): tablet 4838e579469d4b628f30f014c9f2b109 (table test-table [id=968026875ac14b018828f2ed39631347]) successfully deleted
I20260925 10:36:56.443463 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 18138
I20260925 10:36:56.461517 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 18269
I20260925 10:36:56.468765 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.462354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.0.97.2:41227)
0925 10:36:56.462578 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:56.462584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:56.462603 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:36:56.462770 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:56.462775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:56.462868 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:56.462983 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:56.463380 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.464169 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.465422 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.465821 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.465946 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.467331 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:56.467340 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:56.467354 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:36:56.467976 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:56.468018 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:36:56.468036 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:56.468160 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:56.468285 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 10:36:56.468382 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 10:36:56.469588 18391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:36:56.556728 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 18269
I20260925 10:36:56.578035 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 18399
I20260925 10:36:56.586033 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:56.578888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42130 (local address 127.0.97.3:35157)
0925 10:36:56.579074 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:56.579078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:56.579096 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:36:56.579300 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:56.579304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:56.579419 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:56.579517 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:56.580185 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.580990 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.582922 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:56.583348 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:56.583443 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:56.584617 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:56.584626 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:56.584639 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:36:56.585260 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:56.585303 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:36:56.585321 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:56.585404 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:56.585579 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 10:36:56.585648 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:36:56.586933 18521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:36:56.705173 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 18399
I20260925 10:36:56.723991 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 18048
I20260925 10:36:56.724951 18109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:36:56.829908 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 18048
2026-09-25T10:36:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2700 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:36:56 dist-test-slave-dxdd krb5kdc[18576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:36:56 dist-test-slave-dxdd krb5kdc[18576](info): set up 2 sockets
Sep 25 10:36:56 dist-test-slave-dxdd krb5kdc[18576](info): commencing operation
krb5kdc: starting...
W20260925 10:36:57.918282 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:36:58 dist-test-slave-dxdd krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:36:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:36:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:36:58.143712 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44145
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:36:58.456800 18589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:58.457124 18589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:58.457291 18589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:58.464408 18589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:36:58.464537 18589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:58.464635 18589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:58.464740 18589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:36:58.464828 18589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:36:58.473997 18589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:58.475260 18589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:58.476523 18589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:58.488081 18595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:58.488149 18597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:58.488101 18594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:58.488873 18589 server_base.cc:1034] running on GCE node
I20260925 10:36:58.489584 18589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:58.490986 18589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:58.492214 18589 hybrid_clock.cc:648] HybridClock initialized: now 1790332618492177 us; error 48 us; skew 500 ppm
I20260925 10:36:58.492539 18589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:36:58 dist-test-slave-dxdd krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:36:58.495599 18589 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:36:58.496994 18589 webserver.cc:460] Webserver started at http://127.0.97.62:38155/ using document root <none> and password file <none>
I20260925 10:36:58.497632 18589 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:58.497771 18589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:58.498102 18589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:36:58.500942 18589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "0deba845b9424af68b9c4093accd1b04"
format_stamp: "Formatted at 2026-09-25 10:36:58 on dist-test-slave-dxdd"
server_key: "da0757a79f5afaa31594d61b20a8e998"
server_key_iv: "7023bd6b0d78bbf758fd69ce6929249a"
server_key_version: "encryptionkey@0"
I20260925 10:36:58.501684 18589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "0deba845b9424af68b9c4093accd1b04"
format_stamp: "Formatted at 2026-09-25 10:36:58 on dist-test-slave-dxdd"
server_key: "da0757a79f5afaa31594d61b20a8e998"
server_key_iv: "7023bd6b0d78bbf758fd69ce6929249a"
server_key_version: "encryptionkey@0"
I20260925 10:36:58.506711 18589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 10:36:58.509915 18604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:58.511204 18589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:36:58.511377 18589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "0deba845b9424af68b9c4093accd1b04"
format_stamp: "Formatted at 2026-09-25 10:36:58 on dist-test-slave-dxdd"
server_key: "da0757a79f5afaa31594d61b20a8e998"
server_key_iv: "7023bd6b0d78bbf758fd69ce6929249a"
server_key_version: "encryptionkey@0"
I20260925 10:36:58.511570 18589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:58.535184 18589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:58.538817 18589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:58.539090 18589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:58.555872 18589 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44145
I20260925 10:36:58.555907 18655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44145 every 8 connection(s)
I20260925 10:36:58.557106 18589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:36:58.560540 18656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:36:58.564893 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 18589
I20260925 10:36:58.565085 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:36:58.565502 388 external_mini_cluster.cc:1428] Setting key f02d7d8db570d0893fbefc310a82c3b2
I20260925 10:36:58.572172 18656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04: Bootstrap starting.
I20260925 10:36:58.575664 18656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04: Neither blocks nor log segments found. Creating new log.
I20260925 10:36:58.576790 18656 log.cc:824] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04: Log is configured to *not* fsync() on all Append() calls
I20260925 10:36:58.579545 18656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04: No bootstrap required, opened a new log
Sep 25 10:36:58 dist-test-slave-dxdd krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:36:58.588857 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:58.567342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53170 (local address 127.0.97.62:44145)
0925 10:36:58.568127 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:58.568147 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:58.568193 (+ 46us) server_negotiation.cc:407] Connection header received
0925 10:36:58.570695 (+ 2502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:58.570720 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:58.571167 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:58.571589 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:58.572821 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:58.574853 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:58.576198 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:58.577244 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:58.577459 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:58.581924 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:58.581931 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:58.581944 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:58.581974 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:58.585083 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:58.585864 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:58.585869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:58.585874 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:58.586043 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:58.586508 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:58.586512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:58.586513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:58.586995 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:58.587223 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:58.587604 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 10:36:58.587889 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":319,"threads_started":1}
I20260925 10:36:58.588791 18656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0deba845b9424af68b9c4093accd1b04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44145 } }
I20260925 10:36:58.589262 18656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:36:58.589385 18656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0deba845b9424af68b9c4093accd1b04, State: Initialized, Role: FOLLOWER
I20260925 10:36:58.589985 18656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0deba845b9424af68b9c4093accd1b04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44145 } }
I20260925 10:36:58.590274 18656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:36:58.590442 18656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:36:58.590638 18656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:36:58.593160 18656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0deba845b9424af68b9c4093accd1b04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44145 } }
I20260925 10:36:58.593560 18656 leader_election.cc:302] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0deba845b9424af68b9c4093accd1b04; no voters:
I20260925 10:36:58.594033 18656 leader_election.cc:288] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:36:58.594169 18661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:36:58.594905 18661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [term 1 LEADER]: Becoming Leader. State: Replica: 0deba845b9424af68b9c4093accd1b04, State: Running, Role: LEADER
I20260925 10:36:58.595433 18661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0deba845b9424af68b9c4093accd1b04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44145 } }
I20260925 10:36:58.595959 18656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:36:58.598690 18662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0deba845b9424af68b9c4093accd1b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0deba845b9424af68b9c4093accd1b04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44145 } } }
I20260925 10:36:58.598649 18663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0deba845b9424af68b9c4093accd1b04. Latest consensus state: current_term: 1 leader_uuid: "0deba845b9424af68b9c4093accd1b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0deba845b9424af68b9c4093accd1b04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44145 } } }
I20260925 10:36:58.599197 18663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [sys.catalog]: This master's current role is: LEADER
I20260925 10:36:58.599198 18662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04 [sys.catalog]: This master's current role is: LEADER
I20260925 10:36:58.600452 18669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:36:58.605649 18669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:36:58.614296 18669 catalog_manager.cc:1347] Generated new cluster ID: b64db742f31d4c6db5bf08e524d06024
I20260925 10:36:58.614425 18669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:36:58.649385 18669 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:36:58.651069 18669 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:36:58.662756 18669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04: Generated new TSK 0
I20260925 10:36:58.663497 18669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:36:58.729978 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44145
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:36:59.073956 18682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:59.074331 18682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:59.074465 18682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:59.081638 18682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:59.081760 18682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:59.081880 18682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:36:59.091825 18682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:59.093096 18682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:59.094386 18682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:59.107503 18687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:59.107438 18690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:59.107477 18688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:59.108464 18682 server_base.cc:1034] running on GCE node
I20260925 10:36:59.109040 18682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:59.109920 18682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:59.111135 18682 hybrid_clock.cc:648] HybridClock initialized: now 1790332619111089 us; error 59 us; skew 500 ppm
I20260925 10:36:59.111433 18682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:36:59 dist-test-slave-dxdd krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:36:59.114507 18682 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:36:59.116026 18682 webserver.cc:460] Webserver started at http://127.0.97.1:42361/ using document root <none> and password file <none>
I20260925 10:36:59.116704 18682 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:59.116847 18682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:59.117142 18682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:36:59.119987 18682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "47ba04377a49429aad597acb9c92236d"
format_stamp: "Formatted at 2026-09-25 10:36:59 on dist-test-slave-dxdd"
server_key: "470b9cc4f477a7b572e0f4d1958371f8"
server_key_iv: "06d11aa5790b52cb46209d33f3cd783c"
server_key_version: "encryptionkey@0"
I20260925 10:36:59.120756 18682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "47ba04377a49429aad597acb9c92236d"
format_stamp: "Formatted at 2026-09-25 10:36:59 on dist-test-slave-dxdd"
server_key: "470b9cc4f477a7b572e0f4d1958371f8"
server_key_iv: "06d11aa5790b52cb46209d33f3cd783c"
server_key_version: "encryptionkey@0"
I20260925 10:36:59.125670 18682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:36:59.128945 18697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:59.130206 18682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:36:59.130404 18682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "47ba04377a49429aad597acb9c92236d"
format_stamp: "Formatted at 2026-09-25 10:36:59 on dist-test-slave-dxdd"
server_key: "470b9cc4f477a7b572e0f4d1958371f8"
server_key_iv: "06d11aa5790b52cb46209d33f3cd783c"
server_key_version: "encryptionkey@0"
I20260925 10:36:59.130599 18682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:59.153038 18682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:59.157047 18682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:59.157357 18682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:59.158334 18682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:36:59.159626 18682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:36:59.159731 18682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:59.159878 18682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:36:59.159953 18682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:59.192045 18682 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42119
I20260925 10:36:59.192127 18809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42119 every 8 connection(s)
I20260925 10:36:59.193357 18682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:36:59.196271 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 18682
I20260925 10:36:59.196455 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:36:59.197000 388 external_mini_cluster.cc:1428] Setting key 6d21b6eede5d8d9f58cadefbbfa95bd2
Sep 25 10:36:59 dist-test-slave-dxdd krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:36:59.216979 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:59.195852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39025 (local address 127.0.97.62:44145)
0925 10:36:59.196351 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:59.196357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:59.197462 (+ 1105us) server_negotiation.cc:407] Connection header received
0925 10:36:59.199823 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:59.199827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:59.199890 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:59.199975 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:59.202090 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:59.202928 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:59.205343 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:59.205720 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:59.205807 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:59.209879 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:59.209888 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:59.209895 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:59.209936 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:59.213498 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:59.214183 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:59.214192 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:59.214198 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:59.214371 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:59.214842 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:59.214849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:59.214854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:59.215230 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:59.215390 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:59.216222 (+ 832us) server_negotiation.cc:299] Negotiation successful
0925 10:36:59.216503 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":234,"threads_started":1}
I20260925 10:36:59.217633 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:59.196438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44145 (local address 127.0.97.1:39025)
0925 10:36:59.197285 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:59.197340 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:59.199547 (+ 2207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:59.200233 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:59.200247 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:59.201070 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:36:59.201928 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:59.201955 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:59.203068 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:59.203079 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:59.205203 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:59.205221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:59.205941 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:59.205950 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:59.206245 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:59.206965 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:36:59.206999 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:36:59.209672 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:36:59.213637 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:36:59.213645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:36:59.213656 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:36:59.214069 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:36:59.214484 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:36:59.214488 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:36:59.214490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:36:59.214738 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:36:59.215416 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:36:59.215421 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:36:59.215784 (+ 363us) client_negotiation.cc:769] Sending connection context
0925 10:36:59.216068 (+ 284us) client_negotiation.cc:240] Negotiation successful
0925 10:36:59.216513 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":321,"threads_started":1}
I20260925 10:36:59.219244 18810 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44145
I20260925 10:36:59.219571 18810 heartbeater.cc:464] Registering TS with master...
I20260925 10:36:59.220458 18810 heartbeater.cc:511] Master 127.0.97.62:44145 requested a full tablet report, sending...
I20260925 10:36:59.222749 18621 ts_manager.cc:194] Registered new tserver with Master: 47ba04377a49429aad597acb9c92236d (127.0.97.1:42119)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260925 10:36:59.224232 18621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:36:59.253444 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44145
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:36:59.568171 18816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:36:59.568477 18816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:36:59.568616 18816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:36:59.579624 18816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:36:59.579722 18816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:36:59.579834 18816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:36:59.589099 18816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:36:59.590188 18816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:36:59.591403 18816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:36:59.604031 18824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:59.604116 18821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:36:59.604064 18822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:36:59.604745 18816 server_base.cc:1034] running on GCE node
I20260925 10:36:59.605266 18816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:36:59.606139 18816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:36:59.607324 18816 hybrid_clock.cc:648] HybridClock initialized: now 1790332619607267 us; error 50 us; skew 500 ppm
I20260925 10:36:59.607613 18816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:36:59 dist-test-slave-dxdd krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:36:59.610646 18816 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:36:59.611972 18816 webserver.cc:460] Webserver started at http://127.0.97.2:44911/ using document root <none> and password file <none>
I20260925 10:36:59.612556 18816 fs_manager.cc:359] Metadata directory not provided
I20260925 10:36:59.612650 18816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:36:59.612895 18816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:36:59.615694 18816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "02803371e63442b4be8916b58055b395"
format_stamp: "Formatted at 2026-09-25 10:36:59 on dist-test-slave-dxdd"
server_key: "c0a01cac207afd5730e4ef2348d29404"
server_key_iv: "a1838da49eac380fff72e1e795f52103"
server_key_version: "encryptionkey@0"
I20260925 10:36:59.616462 18816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "02803371e63442b4be8916b58055b395"
format_stamp: "Formatted at 2026-09-25 10:36:59 on dist-test-slave-dxdd"
server_key: "c0a01cac207afd5730e4ef2348d29404"
server_key_iv: "a1838da49eac380fff72e1e795f52103"
server_key_version: "encryptionkey@0"
I20260925 10:36:59.621299 18816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:36:59.624578 18831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:59.625787 18816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:36:59.625953 18816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "02803371e63442b4be8916b58055b395"
format_stamp: "Formatted at 2026-09-25 10:36:59 on dist-test-slave-dxdd"
server_key: "c0a01cac207afd5730e4ef2348d29404"
server_key_iv: "a1838da49eac380fff72e1e795f52103"
server_key_version: "encryptionkey@0"
I20260925 10:36:59.626135 18816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:36:59.646198 18816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:36:59.649682 18816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:36:59.649971 18816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:36:59.650915 18816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:36:59.652132 18816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:36:59.652235 18816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:59.652374 18816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:36:59.652443 18816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:36:59.685787 18816 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39839
I20260925 10:36:59.685829 18943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39839 every 8 connection(s)
I20260925 10:36:59.687052 18816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:36:59.695159 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 18816
I20260925 10:36:59.695359 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:36:59.695894 388 external_mini_cluster.cc:1428] Setting key ea8a36860a50d77d1acec50962f8be2e
Sep 25 10:36:59 dist-test-slave-dxdd krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:36:59.706800 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:59.689414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56531 (local address 127.0.97.62:44145)
0925 10:36:59.689589 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:36:59.689593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:36:59.690755 (+ 1162us) server_negotiation.cc:407] Connection header received
0925 10:36:59.692471 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:36:59.692476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:36:59.692559 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:36:59.692678 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:36:59.694120 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:59.694912 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:59.696902 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:36:59.697382 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:36:59.697525 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:59.701117 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:36:59.701122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:36:59.701125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:36:59.701155 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:36:59.703897 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:59.704598 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:59.704608 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:59.704610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:59.704725 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:36:59.705137 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:36:59.705142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:36:59.705144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:36:59.705395 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:36:59.705542 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 10:36:59.706205 (+ 663us) server_negotiation.cc:299] Negotiation successful
0925 10:36:59.706468 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:36:59.707695 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:36:59.689869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44145 (local address 127.0.97.2:56531)
0925 10:36:59.690604 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0925 10:36:59.690640 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:36:59.692215 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:36:59.692902 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:36:59.692910 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:36:59.693416 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:36:59.693952 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:59.693968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:59.695046 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:59.695052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:59.696729 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:36:59.696746 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:36:59.697617 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:36:59.697624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:36:59.697913 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:36:59.698656 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:36:59.698680 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:36:59.700902 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:36:59.704031 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:36:59.704041 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:36:59.704055 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:36:59.704472 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:36:59.704839 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:36:59.704850 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:36:59.704854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:36:59.705024 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:36:59.705524 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:36:59.705532 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:36:59.705833 (+ 301us) client_negotiation.cc:769] Sending connection context
0925 10:36:59.706128 (+ 295us) client_negotiation.cc:240] Negotiation successful
0925 10:36:59.706627 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":277,"threads_started":1}
I20260925 10:36:59.709112 18944 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44145
I20260925 10:36:59.709394 18944 heartbeater.cc:464] Registering TS with master...
I20260925 10:36:59.710093 18944 heartbeater.cc:511] Master 127.0.97.62:44145 requested a full tablet report, sending...
I20260925 10:36:59.711567 18621 ts_manager.cc:194] Registered new tserver with Master: 02803371e63442b4be8916b58055b395 (127.0.97.2:39839)
I20260925 10:36:59.712791 18621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56531
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:36:59.749500 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44145
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:00.062839 18949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:00.063089 18949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:00.063180 18949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:00.069773 18949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:00.069867 18949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:00.070032 18949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:37:00.079046 18949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:00.080228 18949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:00.081418 18949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:00.093986 18954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:00.094156 18955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:00.094241 18957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:00.095129 18949 server_base.cc:1034] running on GCE node
I20260925 10:37:00.095674 18949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:00.096627 18949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:00.097822 18949 hybrid_clock.cc:648] HybridClock initialized: now 1790332620097770 us; error 43 us; skew 500 ppm
I20260925 10:37:00.098106 18949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:00 dist-test-slave-dxdd krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:00.104734 18949 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:37:00.105967 18949 webserver.cc:460] Webserver started at http://127.0.97.3:41711/ using document root <none> and password file <none>
I20260925 10:37:00.106617 18949 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:00.106709 18949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:00.106925 18949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:00.109509 18949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "d58192da08014311b533507e5634a01d"
format_stamp: "Formatted at 2026-09-25 10:37:00 on dist-test-slave-dxdd"
server_key: "af8f461d188d7dc15dd3edc76c1da974"
server_key_iv: "4d4be9b40dc0a4ea530e5a4856a69573"
server_key_version: "encryptionkey@0"
I20260925 10:37:00.110181 18949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d58192da08014311b533507e5634a01d"
format_stamp: "Formatted at 2026-09-25 10:37:00 on dist-test-slave-dxdd"
server_key: "af8f461d188d7dc15dd3edc76c1da974"
server_key_iv: "4d4be9b40dc0a4ea530e5a4856a69573"
server_key_version: "encryptionkey@0"
I20260925 10:37:00.114880 18949 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:37:00.118041 18964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:00.119248 18949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:37:00.119369 18949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "d58192da08014311b533507e5634a01d"
format_stamp: "Formatted at 2026-09-25 10:37:00 on dist-test-slave-dxdd"
server_key: "af8f461d188d7dc15dd3edc76c1da974"
server_key_iv: "4d4be9b40dc0a4ea530e5a4856a69573"
server_key_version: "encryptionkey@0"
I20260925 10:37:00.119509 18949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:00.163646 18949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:00.167402 18949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:00.167631 18949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:00.168581 18949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:37:00.169819 18949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:37:00.169886 18949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:00.169965 18949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:37:00.170008 18949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:00.202585 18949 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36529
I20260925 10:37:00.202627 19076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36529 every 8 connection(s)
I20260925 10:37:00.203754 18949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:37:00.212261 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 18949
I20260925 10:37:00.212433 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:37:00.212921 388 external_mini_cluster.cc:1428] Setting key 85a56c3732a757eb77f9c7ed4637835e
Sep 25 10:37:00 dist-test-slave-dxdd krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:37:00.222000 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.205977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45859 (local address 127.0.97.62:44145)
0925 10:37:00.206177 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:00.206183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:00.207261 (+ 1078us) server_negotiation.cc:407] Connection header received
0925 10:37:00.208896 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:00.208900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:00.208981 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:00.209066 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:00.210441 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.211139 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.212984 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.213496 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.213618 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.217191 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:00.217196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:00.217199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:00.217228 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:00.219599 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:00.220178 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:00.220182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:00.220184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:00.220327 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:00.220670 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:00.220673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:00.220675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:00.220897 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:00.220996 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:00.221558 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 10:37:00.221733 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:37:00.222893 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.206468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44145 (local address 127.0.97.3:45859)
0925 10:37:00.207113 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:00.207146 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:00.208650 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:00.209249 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:00.209256 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:00.209750 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:00.210252 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:00.210268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.211258 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.211262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:00.212830 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:00.212843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.213722 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.213729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:00.213984 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.214815 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:00.214843 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:00.217014 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:00.219715 (+ 2701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:00.219721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:00.219730 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:00.220080 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:00.220425 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:00.220428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:00.220430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:00.220582 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:00.220989 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:00.220994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:00.221262 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 10:37:00.221453 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 10:37:00.221798 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":266,"threads_started":1}
I20260925 10:37:00.224153 19077 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44145
I20260925 10:37:00.224442 19077 heartbeater.cc:464] Registering TS with master...
I20260925 10:37:00.225055 19077 heartbeater.cc:511] Master 127.0.97.62:44145 requested a full tablet report, sending...
I20260925 10:37:00.226512 18621 ts_manager.cc:194] Registered new tserver with Master: d58192da08014311b533507e5634a01d (127.0.97.3:36529)
I20260925 10:37:00.227048 18810 heartbeater.cc:503] Master 127.0.97.62:44145 was elected leader, sending a full tablet report...
I20260925 10:37:00.227515 18621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45859
I20260925 10:37:00.228014 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:37:00.243580 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.232038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.0.97.62:44145)
0925 10:37:00.232238 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:00.232247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:00.232594 (+ 347us) server_negotiation.cc:407] Connection header received
0925 10:37:00.232719 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:00.232723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:00.232773 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:00.232852 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:00.234069 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.234784 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.236060 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.236429 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.236515 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.239013 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:00.239017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:00.239020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:00.239046 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:00.241161 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:00.241825 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:00.241828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:00.241830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:00.241928 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:00.242257 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:00.242260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:00.242262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:00.242571 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:00.242752 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:00.243130 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 10:37:00.243270 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:37:00.247026 18620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:37:00.253604 18620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:37:00.274960 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.264761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.0.97.2:39839)
0925 10:37:00.265337 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:00.265345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:00.265375 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:37:00.265715 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:00.265736 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:00.265993 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:00.266175 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:00.267208 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.269010 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.271935 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.273330 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.273452 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.273649 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:00.274416 (+ 767us) server_negotiation.cc:299] Negotiation successful
0925 10:37:00.274581 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":307,"threads_started":1}
I20260925 10:37:00.275238 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.264692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39839 (local address 127.0.0.1:57208)
0925 10:37:00.265262 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:00.265296 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:00.265449 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:00.266465 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:00.266480 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:00.266517 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:00.266993 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:00.267021 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.269222 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.269229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:00.271780 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:00.271794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.273812 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.273819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:00.274108 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.274202 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 10:37:00.274370 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 10:37:00.274663 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":110,"thread_start_us":641,"threads_started":1}
I20260925 10:37:00.275266 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.266035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36529 (local address 127.0.0.1:53244)
0925 10:37:00.266834 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:00.266856 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:00.267004 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:00.268140 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:00.268144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:00.268172 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:00.268609 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:00.268620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.270172 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.270184 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:00.272261 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:00.272270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.273923 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.273929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:00.274180 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.274203 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:37:00.274369 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 10:37:00.274663 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":125,"thread_start_us":457,"threads_started":1}
I20260925 10:37:00.276294 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.266400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.0.97.3:36529)
0925 10:37:00.267569 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:00.267575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:00.267631 (+ 56us) server_negotiation.cc:407] Connection header received
0925 10:37:00.267739 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:00.267747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:00.267945 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:00.268805 (+ 860us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:00.269063 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.270041 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.272397 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.273718 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.273854 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.274068 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:00.275906 (+ 1838us) server_negotiation.cc:299] Negotiation successful
0925 10:37:00.276042 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":881,"threads_started":1}
I20260925 10:37:00.278555 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.266795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42119 (local address 127.0.0.1:48374)
0925 10:37:00.270753 (+ 3958us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:00.270772 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:00.270877 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:00.271408 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:00.271413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:00.271438 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:00.271852 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:00.271869 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.273628 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.273635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:00.275805 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:00.275818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.277816 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.277821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:00.277975 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.278003 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 10:37:00.278085 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 10:37:00.278197 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3785,"mutex_wait_us":1,"thread_start_us":3585,"threads_started":1}
I20260925 10:37:00.279264 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.267421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.0.97.1:42119)
0925 10:37:00.269105 (+ 1684us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:00.269111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:00.270867 (+ 1756us) server_negotiation.cc:407] Connection header received
0925 10:37:00.271047 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:00.271054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:00.271223 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:00.271408 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:00.272019 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.273486 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.275964 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.277366 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.277487 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.277679 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:00.278824 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0925 10:37:00.278966 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":298,"threads_started":1}
I20260925 10:37:00.279194 18879 tablet_service.cc:1467] Processing CreateTablet for tablet df0d97b4e43d4fa6a6bc2bd280886def (DEFAULT_TABLE table=test-table [id=6f0c7e7bbc6240d2bada83dc243eff86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:37:00.279366 19012 tablet_service.cc:1467] Processing CreateTablet for tablet df0d97b4e43d4fa6a6bc2bd280886def (DEFAULT_TABLE table=test-table [id=6f0c7e7bbc6240d2bada83dc243eff86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:37:00.280536 19012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0d97b4e43d4fa6a6bc2bd280886def. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:00.280643 18879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0d97b4e43d4fa6a6bc2bd280886def. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:00.282855 18745 tablet_service.cc:1467] Processing CreateTablet for tablet df0d97b4e43d4fa6a6bc2bd280886def (DEFAULT_TABLE table=test-table [id=6f0c7e7bbc6240d2bada83dc243eff86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:37:00.284076 18745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0d97b4e43d4fa6a6bc2bd280886def. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:00.292033 19095 tablet_bootstrap.cc:492] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d: Bootstrap starting.
I20260925 10:37:00.295766 19095 tablet_bootstrap.cc:654] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:00.295782 19096 tablet_bootstrap.cc:492] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395: Bootstrap starting.
I20260925 10:37:00.296996 19095 log.cc:824] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:00.297572 19097 tablet_bootstrap.cc:492] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d: Bootstrap starting.
I20260925 10:37:00.299165 19096 tablet_bootstrap.cc:654] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:00.300348 19096 log.cc:824] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:00.300614 19097 tablet_bootstrap.cc:654] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:00.301064 19095 tablet_bootstrap.cc:492] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d: No bootstrap required, opened a new log
I20260925 10:37:00.301321 19095 ts_tablet_manager.cc:1397] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 10:37:00.301923 19097 log.cc:824] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:00.303202 19096 tablet_bootstrap.cc:492] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395: No bootstrap required, opened a new log
I20260925 10:37:00.303459 19096 ts_tablet_manager.cc:1397] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 10:37:00.305115 19097 tablet_bootstrap.cc:492] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d: No bootstrap required, opened a new log
I20260925 10:37:00.305343 19097 ts_tablet_manager.cc:1397] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 10:37:00.309001 19095 raft_consensus.cc:348] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02803371e63442b4be8916b58055b395" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39839 } } peers { permanent_uuid: "d58192da08014311b533507e5634a01d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36529 } } peers { permanent_uuid: "47ba04377a49429aad597acb9c92236d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42119 } }
I20260925 10:37:00.309337 19095 raft_consensus.cc:374] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:00.309442 19095 raft_consensus.cc:729] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58192da08014311b533507e5634a01d, State: Initialized, Role: FOLLOWER
I20260925 10:37:00.310024 19095 consensus_queue.cc:260] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02803371e63442b4be8916b58055b395" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39839 } } peers { permanent_uuid: "d58192da08014311b533507e5634a01d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36529 } } peers { permanent_uuid: "47ba04377a49429aad597acb9c92236d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42119 } }
I20260925 10:37:00.311076 19077 heartbeater.cc:503] Master 127.0.97.62:44145 was elected leader, sending a full tablet report...
I20260925 10:37:00.311635 19095 ts_tablet_manager.cc:1428] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260925 10:37:00.313387 19096 raft_consensus.cc:348] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02803371e63442b4be8916b58055b395" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39839 } } peers { permanent_uuid: "d58192da08014311b533507e5634a01d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36529 } } peers { permanent_uuid: "47ba04377a49429aad597acb9c92236d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42119 } }
I20260925 10:37:00.313793 19096 raft_consensus.cc:374] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:00.313884 19096 raft_consensus.cc:729] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02803371e63442b4be8916b58055b395, State: Initialized, Role: FOLLOWER
I20260925 10:37:00.314559 19096 consensus_queue.cc:260] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02803371e63442b4be8916b58055b395" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39839 } } peers { permanent_uuid: "d58192da08014311b533507e5634a01d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36529 } } peers { permanent_uuid: "47ba04377a49429aad597acb9c92236d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42119 } }
I20260925 10:37:00.315331 19097 raft_consensus.cc:348] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02803371e63442b4be8916b58055b395" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39839 } } peers { permanent_uuid: "d58192da08014311b533507e5634a01d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36529 } } peers { permanent_uuid: "47ba04377a49429aad597acb9c92236d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42119 } }
I20260925 10:37:00.315725 19097 raft_consensus.cc:374] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:00.315837 19097 raft_consensus.cc:729] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47ba04377a49429aad597acb9c92236d, State: Initialized, Role: FOLLOWER
I20260925 10:37:00.316085 18944 heartbeater.cc:503] Master 127.0.97.62:44145 was elected leader, sending a full tablet report...
I20260925 10:37:00.316377 19097 consensus_queue.cc:260] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02803371e63442b4be8916b58055b395" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39839 } } peers { permanent_uuid: "d58192da08014311b533507e5634a01d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36529 } } peers { permanent_uuid: "47ba04377a49429aad597acb9c92236d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42119 } }
I20260925 10:37:00.316692 19096 ts_tablet_manager.cc:1428] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260925 10:37:00.318202 19097 ts_tablet_manager.cc:1428] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260925 10:37:00.439005 18945 tablet.cc:2369] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:37:00.445783 18811 tablet.cc:2369] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:37:00.448118 19102 raft_consensus.cc:482] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:37:00.448390 19102 raft_consensus.cc:504] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02803371e63442b4be8916b58055b395" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39839 } } peers { permanent_uuid: "d58192da08014311b533507e5634a01d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36529 } } peers { permanent_uuid: "47ba04377a49429aad597acb9c92236d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42119 } }
I20260925 10:37:00.449908 19102 leader_election.cc:288] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d58192da08014311b533507e5634a01d (127.0.97.3:36529), 47ba04377a49429aad597acb9c92236d (127.0.97.1:42119)
W20260925 10:37:00.455214 19078 tablet.cc:2369] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:37:00.458149 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.450401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32889 (local address 127.0.97.3:36529)
0925 10:37:00.450579 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:00.450583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:00.451134 (+ 551us) server_negotiation.cc:407] Connection header received
0925 10:37:00.451332 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:00.451339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:00.451420 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:00.451537 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:00.452283 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.453336 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.455922 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.457093 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.457221 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.457319 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:00.457805 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 10:37:00.457921 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:37:00.458154 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.451429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36605 (local address 127.0.97.1:42119)
0925 10:37:00.451633 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:00.451638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:00.451821 (+ 183us) server_negotiation.cc:407] Connection header received
0925 10:37:00.451985 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:00.451988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:00.452040 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:00.452106 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:00.452676 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.453619 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.455921 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.457093 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.457221 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.457317 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:00.457805 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 10:37:00.457921 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 10:37:00.458415 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.451161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42119 (local address 127.0.97.2:36605)
0925 10:37:00.451715 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:00.451732 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:00.451877 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:00.452147 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:00.452151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:00.452176 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:00.452487 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:00.452494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.453770 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.453775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:00.455791 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:00.455800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.457329 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.457340 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:00.457621 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.457671 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 10:37:00.457769 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 10:37:00.457921 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":85,"thread_start_us":297,"threads_started":1}
I20260925 10:37:00.458442 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.450274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36529 (local address 127.0.97.2:32889)
0925 10:37:00.451024 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:00.451044 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:00.451182 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:00.451664 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:00.451669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:00.451693 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:00.452133 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:00.452144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.453470 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.453475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:00.455761 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:00.455776 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.457349 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.457354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:00.457617 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.457676 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 10:37:00.457765 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:37:00.457921 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":136,"spinlock_wait_cycles":31616,"thread_start_us":407,"threads_started":1}
I20260925 10:37:00.459412 18765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0d97b4e43d4fa6a6bc2bd280886def" candidate_uuid: "02803371e63442b4be8916b58055b395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47ba04377a49429aad597acb9c92236d" is_pre_election: true
I20260925 10:37:00.459604 19032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0d97b4e43d4fa6a6bc2bd280886def" candidate_uuid: "02803371e63442b4be8916b58055b395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58192da08014311b533507e5634a01d" is_pre_election: true
I20260925 10:37:00.459903 18765 raft_consensus.cc:2454] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02803371e63442b4be8916b58055b395 in term 0.
I20260925 10:37:00.460081 19032 raft_consensus.cc:2454] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02803371e63442b4be8916b58055b395 in term 0.
I20260925 10:37:00.460511 18833 leader_election.cc:302] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02803371e63442b4be8916b58055b395, 47ba04377a49429aad597acb9c92236d; no voters:
I20260925 10:37:00.460865 19102 raft_consensus.cc:2781] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:37:00.461009 19102 raft_consensus.cc:482] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:37:00.461113 19102 raft_consensus.cc:3037] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:00.463300 19102 raft_consensus.cc:504] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02803371e63442b4be8916b58055b395" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39839 } } peers { permanent_uuid: "d58192da08014311b533507e5634a01d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36529 } } peers { permanent_uuid: "47ba04377a49429aad597acb9c92236d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42119 } }
I20260925 10:37:00.463852 19102 leader_election.cc:288] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [CANDIDATE]: Term 1 election: Requested vote from peers d58192da08014311b533507e5634a01d (127.0.97.3:36529), 47ba04377a49429aad597acb9c92236d (127.0.97.1:42119)
I20260925 10:37:00.464167 19032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0d97b4e43d4fa6a6bc2bd280886def" candidate_uuid: "02803371e63442b4be8916b58055b395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58192da08014311b533507e5634a01d"
I20260925 10:37:00.464259 18765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0d97b4e43d4fa6a6bc2bd280886def" candidate_uuid: "02803371e63442b4be8916b58055b395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47ba04377a49429aad597acb9c92236d"
I20260925 10:37:00.464355 19032 raft_consensus.cc:3037] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:00.464478 18765 raft_consensus.cc:3037] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:00.466595 19032 raft_consensus.cc:2454] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02803371e63442b4be8916b58055b395 in term 1.
I20260925 10:37:00.466728 18765 raft_consensus.cc:2454] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02803371e63442b4be8916b58055b395 in term 1.
I20260925 10:37:00.466962 18833 leader_election.cc:302] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02803371e63442b4be8916b58055b395, d58192da08014311b533507e5634a01d; no voters:
I20260925 10:37:00.467214 19102 raft_consensus.cc:2781] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:00.467625 19102 raft_consensus.cc:686] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [term 1 LEADER]: Becoming Leader. State: Replica: 02803371e63442b4be8916b58055b395, State: Running, Role: LEADER
I20260925 10:37:00.468070 19102 consensus_queue.cc:237] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02803371e63442b4be8916b58055b395" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39839 } } peers { permanent_uuid: "d58192da08014311b533507e5634a01d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36529 } } peers { permanent_uuid: "47ba04377a49429aad597acb9c92236d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42119 } }
I20260925 10:37:00.472172 18620 catalog_manager.cc:5495] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02803371e63442b4be8916b58055b395 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "02803371e63442b4be8916b58055b395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02803371e63442b4be8916b58055b395" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d58192da08014311b533507e5634a01d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47ba04377a49429aad597acb9c92236d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42119 } health_report { overall_health: UNKNOWN } } }
I20260925 10:37:00.511088 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.505762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.0.97.2:39839)
0925 10:37:00.505964 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:00.505970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:00.506040 (+ 70us) server_negotiation.cc:407] Connection header received
0925 10:37:00.506287 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:00.506291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:00.506413 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:00.506561 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:00.506941 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.507657 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.508998 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.509376 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.509523 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.510058 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:00.510561 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:00.510731 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:00.510834 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 10:37:00.510874 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:37:00.518494 18765 raft_consensus.cc:1261] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d [term 1 FOLLOWER]: Refusing update from remote peer 02803371e63442b4be8916b58055b395: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:37:00.518536 19032 raft_consensus.cc:1261] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d [term 1 FOLLOWER]: Refusing update from remote peer 02803371e63442b4be8916b58055b395: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:37:00.519394 19106 consensus_queue.cc:1035] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47ba04377a49429aad597acb9c92236d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:37:00.519769 19102 consensus_queue.cc:1035] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d58192da08014311b533507e5634a01d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:37:00.530490 19110 mvcc.cc:205] Tried to move back new op lower bound from 7333202413634957312 to 7333202413441048576. Current Snapshot: MvccSnapshot[applied={T|T < 7333202413634957312}]
I20260925 10:37:00.539867 19111 mvcc.cc:205] Tried to move back new op lower bound from 7333202413634957312 to 7333202413441048576. Current Snapshot: MvccSnapshot[applied={T|T < 7333202413634957312}]
I20260925 10:37:00.542775 18621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:37:00.543035 18621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:37:00.547012 18621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0deba845b9424af68b9c4093accd1b04: Sending DeleteTablet for 3 replicas of tablet df0d97b4e43d4fa6a6bc2bd280886def
I20260925 10:37:00.548189 19012 tablet_service.cc:1514] Processing DeleteTablet for tablet df0d97b4e43d4fa6a6bc2bd280886def with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:37:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53244
I20260925 10:37:00.548188 18879 tablet_service.cc:1514] Processing DeleteTablet for tablet df0d97b4e43d4fa6a6bc2bd280886def with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:37:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57208
I20260925 10:37:00.548993 19119 tablet_replica.cc:330] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395: stopping tablet replica
I20260925 10:37:00.549407 19119 raft_consensus.cc:2229] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:37:00.550048 19119 raft_consensus.cc:2258] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:37:00.550778 18745 tablet_service.cc:1514] Processing DeleteTablet for tablet df0d97b4e43d4fa6a6bc2bd280886def with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:37:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48374
I20260925 10:37:00.551062 19120 tablet_replica.cc:330] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d: stopping tablet replica
I20260925 10:37:00.551420 19120 raft_consensus.cc:2229] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:37:00.551878 19120 raft_consensus.cc:2258] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:37:00.552582 19119 ts_tablet_manager.cc:1905] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:37:00.552812 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 18682
I20260925 10:37:00.559469 19119 ts_tablet_manager.cc:1918] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:37:00.559698 19119 log.cc:1196] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/df0d97b4e43d4fa6a6bc2bd280886def
I20260925 10:37:00.560252 19119 ts_tablet_manager.cc:1939] T df0d97b4e43d4fa6a6bc2bd280886def P 02803371e63442b4be8916b58055b395: Deleting consensus metadata
I20260925 10:37:00.561805 18606 catalog_manager.cc:4841] TS 02803371e63442b4be8916b58055b395 (127.0.97.2:39839): tablet df0d97b4e43d4fa6a6bc2bd280886def (table test-table [id=6f0c7e7bbc6240d2bada83dc243eff86]) successfully deleted
I20260925 10:37:00.561990 19120 ts_tablet_manager.cc:1905] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:37:00.564322 19121 tablet_replica.cc:330] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d: stopping tablet replica
I20260925 10:37:00.564693 19121 raft_consensus.cc:2229] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:37:00.565135 19121 raft_consensus.cc:2258] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:37:00.567143 19121 ts_tablet_manager.cc:1905] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:37:00.572609 19120 ts_tablet_manager.cc:1918] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:37:00.572767 19120 log.cc:1196] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/df0d97b4e43d4fa6a6bc2bd280886def
I20260925 10:37:00.573272 19120 ts_tablet_manager.cc:1939] T df0d97b4e43d4fa6a6bc2bd280886def P d58192da08014311b533507e5634a01d: Deleting consensus metadata
I20260925 10:37:00.573947 19121 ts_tablet_manager.cc:1918] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:37:00.574093 19121 log.cc:1196] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/df0d97b4e43d4fa6a6bc2bd280886def
I20260925 10:37:00.574735 19121 ts_tablet_manager.cc:1939] T df0d97b4e43d4fa6a6bc2bd280886def P 47ba04377a49429aad597acb9c92236d: Deleting consensus metadata
I20260925 10:37:00.576328 18606 catalog_manager.cc:4841] TS 47ba04377a49429aad597acb9c92236d (127.0.97.1:42119): tablet df0d97b4e43d4fa6a6bc2bd280886def (table test-table [id=6f0c7e7bbc6240d2bada83dc243eff86]) successfully deleted
I20260925 10:37:00.578105 18606 catalog_manager.cc:4841] TS d58192da08014311b533507e5634a01d (127.0.97.3:36529): tablet df0d97b4e43d4fa6a6bc2bd280886def (table test-table [id=6f0c7e7bbc6240d2bada83dc243eff86]) successfully deleted
Sep 25 10:37:00 dist-test-slave-dxdd krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:37:00.589828 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.554573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48376 (local address 127.0.97.1:42119)
0925 10:37:00.554733 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:00.554750 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:00.575343 (+ 20593us) server_negotiation.cc:407] Connection header received
0925 10:37:00.575813 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:00.575819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:00.575895 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:00.575966 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:00.577479 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.578531 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.579786 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.580199 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.580347 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.584386 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:00.584397 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:00.584411 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:00.584442 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:00.587201 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:00.587833 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:00.587849 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:00.587851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:00.587970 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:00.588344 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:00.588350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:00.588353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:00.588676 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:00.588996 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:00.589265 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 10:37:00.589468 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:37:00.590502 18805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:00.681686 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 18682
I20260925 10:37:00.704183 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 18816
Sep 25 10:37:00 dist-test-slave-dxdd krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:37:00.723995 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.704975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.0.97.2:39839)
0925 10:37:00.705174 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:00.705180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:00.705258 (+ 78us) server_negotiation.cc:407] Connection header received
0925 10:37:00.705486 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:00.705493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:00.705599 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:00.705708 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:00.707394 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.708539 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.710485 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.711172 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.711298 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.716824 (+ 5526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:00.716838 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:00.716856 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:00.716902 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:00.721020 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:00.721700 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:00.721735 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:00.721739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:00.721901 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:00.722298 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:00.722321 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:00.722325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:00.722765 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:00.723170 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:00.723396 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 10:37:00.723609 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 10:37:00.724723 18939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:00.839402 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 18816
I20260925 10:37:00.857393 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 18949
Sep 25 10:37:00 dist-test-slave-dxdd krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:37:00.871969 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:00.858153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53256 (local address 127.0.97.3:36529)
0925 10:37:00.858365 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:00.858371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:00.858464 (+ 93us) server_negotiation.cc:407] Connection header received
0925 10:37:00.858666 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:00.858675 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:00.858737 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:00.858837 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:00.860021 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.860736 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.862038 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:00.862493 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:00.862620 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:00.866867 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:00.866878 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:00.866893 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:00.866926 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:00.869299 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:00.869959 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:00.869966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:00.869969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:00.870101 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:00.870516 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:00.870521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:00.870524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:00.870861 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:00.871172 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:00.871456 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 10:37:00.871667 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:37:00.872599 19072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:00.959002 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 18949
I20260925 10:37:00.975272 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 18589
I20260925 10:37:00.976271 18651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:01.085992 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 18589
2026-09-25T10:37:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4257 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T10:37:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:01Z Disabled control of system clock
I20260925 10:37:01.157618 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45557
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:37:01.473562 19130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:01.473819 19130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:01.473898 19130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:01.480940 19130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:01.481032 19130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:01.481088 19130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:01.481169 19130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:01.481213 19130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:01.490089 19130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45557
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:01.491291 19130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:01.492902 19130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:01.504477 19135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:01.504632 19136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:01.504765 19138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:01.505544 19130 server_base.cc:1034] running on GCE node
I20260925 10:37:01.506245 19130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:01.507622 19130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:01.508852 19130 hybrid_clock.cc:648] HybridClock initialized: now 1790332621508822 us; error 48 us; skew 500 ppm
I20260925 10:37:01.509219 19130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:37:01.511447 19130 webserver.cc:460] Webserver started at http://127.0.97.62:34471/ using document root <none> and password file <none>
I20260925 10:37:01.512121 19130 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:01.512251 19130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:01.512562 19130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:01.515437 19130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "56d212bc5dfa4e0096ebdfc987851818"
format_stamp: "Formatted at 2026-09-25 10:37:01 on dist-test-slave-dxdd"
server_key: "b82b8851b72d5efa570510254509c12d"
server_key_iv: "e6239f29dfbae7d7637a902c0a80a35a"
server_key_version: "encryptionkey@0"
I20260925 10:37:01.516222 19130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "56d212bc5dfa4e0096ebdfc987851818"
format_stamp: "Formatted at 2026-09-25 10:37:01 on dist-test-slave-dxdd"
server_key: "b82b8851b72d5efa570510254509c12d"
server_key_iv: "e6239f29dfbae7d7637a902c0a80a35a"
server_key_version: "encryptionkey@0"
I20260925 10:37:01.521181 19130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260925 10:37:01.524447 19144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:01.525626 19130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:37:01.525803 19130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "56d212bc5dfa4e0096ebdfc987851818"
format_stamp: "Formatted at 2026-09-25 10:37:01 on dist-test-slave-dxdd"
server_key: "b82b8851b72d5efa570510254509c12d"
server_key_iv: "e6239f29dfbae7d7637a902c0a80a35a"
server_key_version: "encryptionkey@0"
I20260925 10:37:01.526047 19130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:01.537678 19130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:01.538492 19130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:01.538763 19130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:01.555712 19130 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45557
I20260925 10:37:01.555732 19195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45557 every 8 connection(s)
I20260925 10:37:01.556954 19130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:01.558295 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 19130
I20260925 10:37:01.558485 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:01.559055 388 external_mini_cluster.cc:1428] Setting key 9201a27b9d0774d07d2f3a0f6f23eb07
I20260925 10:37:01.561442 19196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:01.572013 19199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:01.561073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41822 (local address 127.0.97.62:45557)
0925 10:37:01.562024 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:01.562034 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:01.562071 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:37:01.563702 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:01.563722 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:01.564159 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:01.564536 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:01.565068 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:01.566435 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:01.567767 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:01.568405 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:01.568589 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:01.569771 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:01.569778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:01.569791 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:01.570398 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:01.570470 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:01.570489 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:01.570618 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:01.570833 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 10:37:01.571110 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":360,"threads_started":1}
I20260925 10:37:01.572304 19196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818: Bootstrap starting.
I20260925 10:37:01.575769 19196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:01.577013 19196 log.cc:824] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:01.579556 19196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818: No bootstrap required, opened a new log
I20260925 10:37:01.586800 19196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d212bc5dfa4e0096ebdfc987851818" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45557 } }
I20260925 10:37:01.587241 19196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:01.587324 19196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d212bc5dfa4e0096ebdfc987851818, State: Initialized, Role: FOLLOWER
I20260925 10:37:01.587854 19196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d212bc5dfa4e0096ebdfc987851818" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45557 } }
I20260925 10:37:01.588044 19196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:01.588143 19196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:01.588267 19196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:01.590224 19196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d212bc5dfa4e0096ebdfc987851818" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45557 } }
I20260925 10:37:01.590869 19196 leader_election.cc:302] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d212bc5dfa4e0096ebdfc987851818; no voters:
I20260925 10:37:01.591442 19196 leader_election.cc:288] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:01.591583 19201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:01.592268 19201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [term 1 LEADER]: Becoming Leader. State: Replica: 56d212bc5dfa4e0096ebdfc987851818, State: Running, Role: LEADER
I20260925 10:37:01.592796 19201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d212bc5dfa4e0096ebdfc987851818" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45557 } }
I20260925 10:37:01.593444 19196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:01.596089 19203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56d212bc5dfa4e0096ebdfc987851818. Latest consensus state: current_term: 1 leader_uuid: "56d212bc5dfa4e0096ebdfc987851818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d212bc5dfa4e0096ebdfc987851818" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45557 } } }
I20260925 10:37:01.596552 19203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:01.595669 19202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56d212bc5dfa4e0096ebdfc987851818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d212bc5dfa4e0096ebdfc987851818" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45557 } } }
I20260925 10:37:01.597447 19202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:01.599211 19206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:01.603128 19206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:01.611440 19206 catalog_manager.cc:1347] Generated new cluster ID: b112ce08c074420eaa76caaa6819b958
I20260925 10:37:01.611537 19206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:01.636936 19206 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:01.638175 19206 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:01.646695 19206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818: Generated new TSK 0
I20260925 10:37:01.647738 19206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:01.653323 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45557
--builtin_ntp_servers=127.0.97.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:37:01.978071 19220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:01.978340 19220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:01.978472 19220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:01.985319 19220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:01.985407 19220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:01.985512 19220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:37:01.994339 19220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:01.995559 19220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:01.997215 19220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:02.009691 19225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:02.009781 19226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:02.009734 19228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:02.010834 19220 server_base.cc:1034] running on GCE node
I20260925 10:37:02.011337 19220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:02.012187 19220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:02.013381 19220 hybrid_clock.cc:648] HybridClock initialized: now 1790332622013334 us; error 36 us; skew 500 ppm
I20260925 10:37:02.013645 19220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:37:02.015738 19220 webserver.cc:460] Webserver started at http://127.0.97.1:41085/ using document root <none> and password file <none>
I20260925 10:37:02.016340 19220 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:02.016419 19220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:02.016675 19220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:02.019461 19220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "653139e7d9d14383b9cec643ac6f8e2e"
format_stamp: "Formatted at 2026-09-25 10:37:02 on dist-test-slave-dxdd"
server_key: "74a6136856996cdec6a49940d439c390"
server_key_iv: "3f3ffe5fb35a57dbe4bf6f4dcf540f83"
server_key_version: "encryptionkey@0"
I20260925 10:37:02.020162 19220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "653139e7d9d14383b9cec643ac6f8e2e"
format_stamp: "Formatted at 2026-09-25 10:37:02 on dist-test-slave-dxdd"
server_key: "74a6136856996cdec6a49940d439c390"
server_key_iv: "3f3ffe5fb35a57dbe4bf6f4dcf540f83"
server_key_version: "encryptionkey@0"
I20260925 10:37:02.025049 19220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 10:37:02.028497 19234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:02.030000 19220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:37:02.030184 19220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "653139e7d9d14383b9cec643ac6f8e2e"
format_stamp: "Formatted at 2026-09-25 10:37:02 on dist-test-slave-dxdd"
server_key: "74a6136856996cdec6a49940d439c390"
server_key_iv: "3f3ffe5fb35a57dbe4bf6f4dcf540f83"
server_key_version: "encryptionkey@0"
I20260925 10:37:02.030465 19220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:02.061046 19220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:02.061944 19220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:02.062224 19220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:02.063316 19220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:37:02.064785 19220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:37:02.064910 19220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:02.065048 19220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:37:02.065140 19220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:02.098125 19220 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39465
I20260925 10:37:02.098212 19346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39465 every 8 connection(s)
I20260925 10:37:02.099552 19220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:37:02.109382 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 19220
I20260925 10:37:02.109555 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:37:02.110153 388 external_mini_cluster.cc:1428] Setting key 5e8c39427cb346f4ec8eb36afe13e9ba
I20260925 10:37:02.112172 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:02.102135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45155 (local address 127.0.97.62:45557)
0925 10:37:02.102600 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:02.102604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:02.103819 (+ 1215us) server_negotiation.cc:407] Connection header received
0925 10:37:02.105631 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:02.105637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:02.105723 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:02.105841 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:02.106898 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:02.107587 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:02.109162 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:02.109680 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:02.109781 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:02.111356 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:02.111361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:02.111365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:02.111385 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:02.111429 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:02.111443 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:02.111535 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:02.111741 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 10:37:02.111830 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":258,"threads_started":1}
I20260925 10:37:02.113193 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45557
--builtin_ntp_servers=127.0.97.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:37:02.113343 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:02.102782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45557 (local address 127.0.97.1:45155)
0925 10:37:02.103667 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:02.103707 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:02.105333 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:02.106064 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:02.106072 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:02.106152 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:02.106740 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:02.106756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:02.107750 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:02.107760 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:02.108981 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:02.108998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:02.109900 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:02.109908 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:02.110206 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:02.111015 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:37:02.111018 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:02.111070 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:37:02.111160 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:02.111539 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:02.111547 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:02.111553 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:37:02.111684 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 10:37:02.112051 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":279,"threads_started":1}
I20260925 10:37:02.114800 19347 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45557
I20260925 10:37:02.115098 19347 heartbeater.cc:464] Registering TS with master...
I20260925 10:37:02.115971 19347 heartbeater.cc:511] Master 127.0.97.62:45557 requested a full tablet report, sending...
I20260925 10:37:02.117988 19161 ts_manager.cc:194] Registered new tserver with Master: 653139e7d9d14383b9cec643ac6f8e2e (127.0.97.1:39465)
I20260925 10:37:02.119541 19161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:45155
W20260925 10:37:02.432983 19351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:02.433274 19351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:02.433418 19351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:02.440397 19351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:02.440526 19351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:02.440711 19351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:37:02.449386 19351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:02.450572 19351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:02.452308 19351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:02.464504 19356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:02.464452 19357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:02.464684 19359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:02.465147 19351 server_base.cc:1034] running on GCE node
I20260925 10:37:02.465761 19351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:02.466672 19351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:02.467855 19351 hybrid_clock.cc:648] HybridClock initialized: now 1790332622467801 us; error 47 us; skew 500 ppm
I20260925 10:37:02.468137 19351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:37:02.470116 19351 webserver.cc:460] Webserver started at http://127.0.97.2:44269/ using document root <none> and password file <none>
I20260925 10:37:02.470801 19351 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:02.470911 19351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:02.471220 19351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:02.473783 19351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "80df0323e95e4a29b2fb86d071083e7b"
format_stamp: "Formatted at 2026-09-25 10:37:02 on dist-test-slave-dxdd"
server_key: "ce40ce7879b64b0d5c164c54e2fe6eda"
server_key_iv: "41239c1ad3ecad21c5a57dca5e4b8bb7"
server_key_version: "encryptionkey@0"
I20260925 10:37:02.474583 19351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "80df0323e95e4a29b2fb86d071083e7b"
format_stamp: "Formatted at 2026-09-25 10:37:02 on dist-test-slave-dxdd"
server_key: "ce40ce7879b64b0d5c164c54e2fe6eda"
server_key_iv: "41239c1ad3ecad21c5a57dca5e4b8bb7"
server_key_version: "encryptionkey@0"
I20260925 10:37:02.479082 19351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:37:02.482175 19365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:02.483379 19351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:37:02.483547 19351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "80df0323e95e4a29b2fb86d071083e7b"
format_stamp: "Formatted at 2026-09-25 10:37:02 on dist-test-slave-dxdd"
server_key: "ce40ce7879b64b0d5c164c54e2fe6eda"
server_key_iv: "41239c1ad3ecad21c5a57dca5e4b8bb7"
server_key_version: "encryptionkey@0"
I20260925 10:37:02.483752 19351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:02.509306 19351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:02.510003 19351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:02.510253 19351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:02.511113 19351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:37:02.512405 19351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:37:02.512532 19351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:02.512648 19351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:37:02.512729 19351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:02.544284 19351 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37327
I20260925 10:37:02.544361 19477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37327 every 8 connection(s)
I20260925 10:37:02.545441 19351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:37:02.547015 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 19351
I20260925 10:37:02.547179 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:37:02.547658 388 external_mini_cluster.cc:1428] Setting key e46ae452539c6127763c667ec8d444f0
I20260925 10:37:02.550201 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45557
--builtin_ntp_servers=127.0.97.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:37:02.557848 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:02.547830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45767 (local address 127.0.97.62:45557)
0925 10:37:02.548026 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:02.548032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:02.549546 (+ 1514us) server_negotiation.cc:407] Connection header received
0925 10:37:02.551335 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:02.551343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:02.551431 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:02.551542 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:02.552487 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:02.553570 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:02.555106 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:02.555638 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:02.555771 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:02.557049 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:02.557055 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:02.557059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:02.557084 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:02.557127 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:02.557138 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:02.557228 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:02.557445 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 10:37:02.557538 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:37:02.559108 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:02.548397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45557 (local address 127.0.97.2:45767)
0925 10:37:02.549386 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:02.549433 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:02.551080 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:02.551715 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:02.551722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:02.551801 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:02.552321 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:02.552339 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:02.553711 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:02.553715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:02.554958 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:02.554969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:02.555849 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:02.555856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:02.556074 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:02.556754 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:37:02.556756 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:02.556795 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:37:02.556859 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:02.557242 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:02.557256 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:02.557267 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 10:37:02.557431 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 10:37:02.557782 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":333,"threads_started":1}
I20260925 10:37:02.560469 19478 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45557
I20260925 10:37:02.560734 19478 heartbeater.cc:464] Registering TS with master...
I20260925 10:37:02.561388 19478 heartbeater.cc:511] Master 127.0.97.62:45557 requested a full tablet report, sending...
I20260925 10:37:02.562702 19161 ts_manager.cc:194] Registered new tserver with Master: 80df0323e95e4a29b2fb86d071083e7b (127.0.97.2:37327)
I20260925 10:37:02.563484 19161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:45767
W20260925 10:37:02.875015 19481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:02.875259 19481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:02.875396 19481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:02.882634 19481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:02.882728 19481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:02.882947 19481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:37:02.892119 19481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:02.893357 19481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:02.895206 19481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:02.907869 19489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:02.907992 19487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:02.908042 19486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:02.908979 19481 server_base.cc:1034] running on GCE node
I20260925 10:37:02.909543 19481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:02.910470 19481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:02.911691 19481 hybrid_clock.cc:648] HybridClock initialized: now 1790332622911645 us; error 38 us; skew 500 ppm
I20260925 10:37:02.911996 19481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:37:02.914251 19481 webserver.cc:460] Webserver started at http://127.0.97.3:36611/ using document root <none> and password file <none>
I20260925 10:37:02.914933 19481 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:02.915061 19481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:02.915375 19481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:02.918092 19481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "7f8ddaaee44047a7933126f112ead5aa"
format_stamp: "Formatted at 2026-09-25 10:37:02 on dist-test-slave-dxdd"
server_key: "40a6c34035c96be7fbc6dca87310b3f0"
server_key_iv: "1e1b4fd2377824972546957390d8dbd7"
server_key_version: "encryptionkey@0"
I20260925 10:37:02.918916 19481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7f8ddaaee44047a7933126f112ead5aa"
format_stamp: "Formatted at 2026-09-25 10:37:02 on dist-test-slave-dxdd"
server_key: "40a6c34035c96be7fbc6dca87310b3f0"
server_key_iv: "1e1b4fd2377824972546957390d8dbd7"
server_key_version: "encryptionkey@0"
I20260925 10:37:02.923749 19481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:37:02.927158 19495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:02.928467 19481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 10:37:02.928649 19481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "7f8ddaaee44047a7933126f112ead5aa"
format_stamp: "Formatted at 2026-09-25 10:37:02 on dist-test-slave-dxdd"
server_key: "40a6c34035c96be7fbc6dca87310b3f0"
server_key_iv: "1e1b4fd2377824972546957390d8dbd7"
server_key_version: "encryptionkey@0"
I20260925 10:37:02.928869 19481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:02.946930 19481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:02.947901 19481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:02.948190 19481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:02.949134 19481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:37:02.950695 19481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:37:02.950825 19481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:02.950968 19481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:37:02.951056 19481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:02.982817 19481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45995
I20260925 10:37:02.982869 19607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45995 every 8 connection(s)
I20260925 10:37:02.984086 19481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:37:02.990391 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 19481
I20260925 10:37:02.990582 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:37:02.991102 388 external_mini_cluster.cc:1428] Setting key 6a8ce96a1fe341cdd1ecf682593a99da
I20260925 10:37:02.996435 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:02.986502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42565 (local address 127.0.97.62:45557)
0925 10:37:02.986696 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:02.986700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:02.988102 (+ 1402us) server_negotiation.cc:407] Connection header received
0925 10:37:02.989782 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:02.989786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:02.989853 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:02.989951 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:02.991321 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:02.992079 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:02.993634 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:02.994130 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:02.994257 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:02.995647 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:02.995652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:02.995655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:02.995672 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:02.995700 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:02.995712 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:02.995805 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:02.996022 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 10:37:02.996109 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 10:37:02.997287 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:02.986984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45557 (local address 127.0.97.3:42565)
0925 10:37:02.987944 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:02.987992 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:02.989525 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:02.990214 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:02.990225 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:02.990353 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:02.991139 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:02.991166 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:02.992227 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:02.992233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:02.993492 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:02.993503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:02.994372 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:02.994382 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:02.994664 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:02.995344 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:37:02.995346 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:02.995382 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:37:02.995459 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:02.995805 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:02.995818 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:02.995827 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:37:02.995992 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 10:37:02.996339 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":349,"threads_started":1}
I20260925 10:37:02.998525 19608 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45557
I20260925 10:37:02.998816 19608 heartbeater.cc:464] Registering TS with master...
I20260925 10:37:02.999491 19608 heartbeater.cc:511] Master 127.0.97.62:45557 requested a full tablet report, sending...
I20260925 10:37:03.001145 19161 ts_manager.cc:194] Registered new tserver with Master: 7f8ddaaee44047a7933126f112ead5aa (127.0.97.3:45995)
I20260925 10:37:03.001919 19161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:42565
I20260925 10:37:03.005626 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:37:03.014685 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.009390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.0.97.62:45557)
0925 10:37:03.009599 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:03.009608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:03.009932 (+ 324us) server_negotiation.cc:407] Connection header received
0925 10:37:03.010060 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:03.010064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:03.010118 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:03.010194 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:03.010668 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.011325 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.012605 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.012958 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.013047 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:03.013980 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:03.013984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:03.013987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:03.014004 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:03.014032 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:03.014042 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:03.014141 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:03.014339 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 10:37:03.014419 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:37:03.017925 19161 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:37:03.024497 19161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:37:03.044579 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.035735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.0.97.3:45995)
0925 10:37:03.036329 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:03.036338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:03.036372 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:37:03.036657 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:03.036680 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:03.036932 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:03.037123 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:03.038255 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.040002 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.042034 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.043488 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.043647 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:03.043913 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:03.044117 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 10:37:03.044264 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":738,"threads_started":1}
I20260925 10:37:03.044574 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.035608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45995 (local address 127.0.0.1:39092)
0925 10:37:03.036268 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:03.036312 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:03.036465 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:03.037480 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:03.037491 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:03.037526 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:03.038045 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:03.038060 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.040131 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.040138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:03.041880 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:03.041890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.043673 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.043679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:03.043867 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:03.043918 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 10:37:03.044081 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 10:37:03.044250 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":282,"threads_started":1}
I20260925 10:37:03.049252 19543 tablet_service.cc:1467] Processing CreateTablet for tablet be72f30a0b844371a7612cac82be283f (DEFAULT_TABLE table=test-table [id=07b02e0124a04c2697da1a6f0692cab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:37:03.050823 19543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be72f30a0b844371a7612cac82be283f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:03.056844 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.037851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39465 (local address 127.0.0.1:41084)
0925 10:37:03.038746 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:03.038765 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:03.038888 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:03.040829 (+ 1941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:03.040834 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:03.040859 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:03.041243 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:03.041256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.044289 (+ 3033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.044297 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:03.052987 (+ 8690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:03.053004 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.056123 (+ 3119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.056129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:03.056308 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:03.056343 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:37:03.056407 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 10:37:03.056532 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":1264,"threads_started":1}
I20260925 10:37:03.057989 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.038335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.0.97.1:39465)
0925 10:37:03.040144 (+ 1809us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:03.040155 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:03.040191 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:37:03.040318 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:03.040328 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:03.040613 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:03.040842 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:03.041433 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.044090 (+ 2657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.053346 (+ 9256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.054864 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.057203 (+ 2339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:03.057439 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:03.057513 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 10:37:03.057661 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1603,"thread_start_us":1328,"threads_started":1}
I20260925 10:37:03.061317 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.037143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37327 (local address 127.0.0.1:55916)
0925 10:37:03.037925 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:03.037951 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:03.038104 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:03.054151 (+ 16047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:03.054156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:03.054181 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:03.054505 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:03.054522 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.057391 (+ 2869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.057396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:03.058845 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:03.058860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.060585 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.060591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:03.060749 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:03.060779 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 10:37:03.060858 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:37:03.060970 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":347,"threads_started":1}
I20260925 10:37:03.061137 19282 tablet_service.cc:1467] Processing CreateTablet for tablet be72f30a0b844371a7612cac82be283f (DEFAULT_TABLE table=test-table [id=07b02e0124a04c2697da1a6f0692cab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:37:03.062422 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.037277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.0.97.2:37327)
0925 10:37:03.052582 (+ 15305us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:03.052591 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:03.052620 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:37:03.052712 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:03.052719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:03.052942 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:03.053130 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:03.055170 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.057234 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.059013 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.060392 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.060534 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:03.061810 (+ 1276us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:03.061885 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 10:37:03.062061 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15132,"thread_start_us":16055,"threads_started":1}
I20260925 10:37:03.062568 19282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be72f30a0b844371a7612cac82be283f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:03.064812 19626 tablet_bootstrap.cc:492] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa: Bootstrap starting.
I20260925 10:37:03.065644 19413 tablet_service.cc:1467] Processing CreateTablet for tablet be72f30a0b844371a7612cac82be283f (DEFAULT_TABLE table=test-table [id=07b02e0124a04c2697da1a6f0692cab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:37:03.066944 19413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be72f30a0b844371a7612cac82be283f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:03.067951 19626 tablet_bootstrap.cc:654] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:03.069160 19626 log.cc:824] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:03.072340 19626 tablet_bootstrap.cc:492] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa: No bootstrap required, opened a new log
I20260925 10:37:03.072594 19626 ts_tablet_manager.cc:1397] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260925 10:37:03.078182 19628 tablet_bootstrap.cc:492] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e: Bootstrap starting.
I20260925 10:37:03.080668 19629 tablet_bootstrap.cc:492] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b: Bootstrap starting.
I20260925 10:37:03.080816 19626 raft_consensus.cc:348] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8ddaaee44047a7933126f112ead5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "80df0323e95e4a29b2fb86d071083e7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } } peers { permanent_uuid: "653139e7d9d14383b9cec643ac6f8e2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39465 } }
I20260925 10:37:03.081347 19626 raft_consensus.cc:374] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:03.081501 19626 raft_consensus.cc:729] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8ddaaee44047a7933126f112ead5aa, State: Initialized, Role: FOLLOWER
I20260925 10:37:03.081861 19628 tablet_bootstrap.cc:654] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:03.082185 19626 consensus_queue.cc:260] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8ddaaee44047a7933126f112ead5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "80df0323e95e4a29b2fb86d071083e7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } } peers { permanent_uuid: "653139e7d9d14383b9cec643ac6f8e2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39465 } }
I20260925 10:37:03.083222 19628 log.cc:824] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:03.083348 19608 heartbeater.cc:503] Master 127.0.97.62:45557 was elected leader, sending a full tablet report...
I20260925 10:37:03.084117 19626 ts_tablet_manager.cc:1428] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260925 10:37:03.084344 19629 tablet_bootstrap.cc:654] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:03.085532 19629 log.cc:824] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:03.087314 19628 tablet_bootstrap.cc:492] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e: No bootstrap required, opened a new log
I20260925 10:37:03.087600 19628 ts_tablet_manager.cc:1397] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260925 10:37:03.088389 19629 tablet_bootstrap.cc:492] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b: No bootstrap required, opened a new log
I20260925 10:37:03.088637 19629 ts_tablet_manager.cc:1397] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 10:37:03.094998 19628 raft_consensus.cc:348] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8ddaaee44047a7933126f112ead5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "80df0323e95e4a29b2fb86d071083e7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } } peers { permanent_uuid: "653139e7d9d14383b9cec643ac6f8e2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39465 } }
I20260925 10:37:03.095331 19628 raft_consensus.cc:374] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:03.095433 19628 raft_consensus.cc:729] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653139e7d9d14383b9cec643ac6f8e2e, State: Initialized, Role: FOLLOWER
I20260925 10:37:03.095397 19629 raft_consensus.cc:348] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8ddaaee44047a7933126f112ead5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "80df0323e95e4a29b2fb86d071083e7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } } peers { permanent_uuid: "653139e7d9d14383b9cec643ac6f8e2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39465 } }
I20260925 10:37:03.095786 19629 raft_consensus.cc:374] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:03.095892 19629 raft_consensus.cc:729] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80df0323e95e4a29b2fb86d071083e7b, State: Initialized, Role: FOLLOWER
I20260925 10:37:03.096040 19628 consensus_queue.cc:260] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8ddaaee44047a7933126f112ead5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "80df0323e95e4a29b2fb86d071083e7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } } peers { permanent_uuid: "653139e7d9d14383b9cec643ac6f8e2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39465 } }
I20260925 10:37:03.096378 19629 consensus_queue.cc:260] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8ddaaee44047a7933126f112ead5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "80df0323e95e4a29b2fb86d071083e7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } } peers { permanent_uuid: "653139e7d9d14383b9cec643ac6f8e2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39465 } }
I20260925 10:37:03.097155 19347 heartbeater.cc:503] Master 127.0.97.62:45557 was elected leader, sending a full tablet report...
I20260925 10:37:03.097389 19478 heartbeater.cc:503] Master 127.0.97.62:45557 was elected leader, sending a full tablet report...
I20260925 10:37:03.097895 19628 ts_tablet_manager.cc:1428] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
W20260925 10:37:03.101869 19348 tablet.cc:2369] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:37:03.109422 19629 ts_tablet_manager.cc:1428] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b: Time spent starting tablet: real 0.021s user 0.003s sys 0.005s
I20260925 10:37:03.183898 19634 raft_consensus.cc:482] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:37:03.184196 19634 raft_consensus.cc:504] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8ddaaee44047a7933126f112ead5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "80df0323e95e4a29b2fb86d071083e7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } } peers { permanent_uuid: "653139e7d9d14383b9cec643ac6f8e2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39465 } }
I20260925 10:37:03.185712 19634 leader_election.cc:288] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f8ddaaee44047a7933126f112ead5aa (127.0.97.3:45995), 653139e7d9d14383b9cec643ac6f8e2e (127.0.97.1:39465)
I20260925 10:37:03.194738 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.186256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53521 (local address 127.0.97.3:45995)
0925 10:37:03.186521 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:03.186529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:03.187022 (+ 493us) server_negotiation.cc:407] Connection header received
0925 10:37:03.187252 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:03.187258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:03.187339 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:03.187455 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:03.188221 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.189282 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.192337 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.193610 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.193715 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:03.193802 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:03.194356 (+ 554us) server_negotiation.cc:299] Negotiation successful
0925 10:37:03.194490 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 10:37:03.194741 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.186243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37999 (local address 127.0.97.1:39465)
0925 10:37:03.186497 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:03.186506 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:03.187125 (+ 619us) server_negotiation.cc:407] Connection header received
0925 10:37:03.187273 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:03.187277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:03.187355 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:03.187447 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:03.188201 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.189282 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.192334 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.193536 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.193677 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:03.193765 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:03.194356 (+ 591us) server_negotiation.cc:299] Negotiation successful
0925 10:37:03.194485 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:37:03.194846 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.186000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45995 (local address 127.0.97.2:53521)
0925 10:37:03.186891 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:03.186912 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:03.187086 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:03.187487 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:03.187492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:03.187518 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:03.188054 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:03.188063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.189413 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.189418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:03.192119 (+ 2701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:03.192130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.193762 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.193767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:03.194029 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:03.194135 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 10:37:03.194284 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 10:37:03.194471 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":283,"thread_start_us":519,"threads_started":1}
I20260925 10:37:03.194897 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.186097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39465 (local address 127.0.97.2:37999)
0925 10:37:03.187018 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:03.187032 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:03.187150 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:03.187487 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:03.187492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:03.187577 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:03.188027 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:03.188046 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.189416 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.189419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:03.192115 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:03.192130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.193734 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.193743 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:03.194023 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:03.194133 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 10:37:03.194284 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 10:37:03.194492 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":288,"spinlock_wait_cycles":20864,"thread_start_us":597,"threads_started":1}
I20260925 10:37:03.196200 19563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be72f30a0b844371a7612cac82be283f" candidate_uuid: "80df0323e95e4a29b2fb86d071083e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8ddaaee44047a7933126f112ead5aa" is_pre_election: true
I20260925 10:37:03.196199 19302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be72f30a0b844371a7612cac82be283f" candidate_uuid: "80df0323e95e4a29b2fb86d071083e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653139e7d9d14383b9cec643ac6f8e2e" is_pre_election: true
I20260925 10:37:03.196772 19563 raft_consensus.cc:2454] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80df0323e95e4a29b2fb86d071083e7b in term 0.
I20260925 10:37:03.196774 19302 raft_consensus.cc:2454] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80df0323e95e4a29b2fb86d071083e7b in term 0.
I20260925 10:37:03.197485 19366 leader_election.cc:302] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f8ddaaee44047a7933126f112ead5aa, 80df0323e95e4a29b2fb86d071083e7b; no voters:
I20260925 10:37:03.197887 19634 raft_consensus.cc:2781] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:37:03.198047 19634 raft_consensus.cc:482] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:37:03.198153 19634 raft_consensus.cc:3037] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:03.200284 19634 raft_consensus.cc:504] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8ddaaee44047a7933126f112ead5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "80df0323e95e4a29b2fb86d071083e7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } } peers { permanent_uuid: "653139e7d9d14383b9cec643ac6f8e2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39465 } }
I20260925 10:37:03.200826 19634 leader_election.cc:288] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [CANDIDATE]: Term 1 election: Requested vote from peers 7f8ddaaee44047a7933126f112ead5aa (127.0.97.3:45995), 653139e7d9d14383b9cec643ac6f8e2e (127.0.97.1:39465)
I20260925 10:37:03.201277 19563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be72f30a0b844371a7612cac82be283f" candidate_uuid: "80df0323e95e4a29b2fb86d071083e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8ddaaee44047a7933126f112ead5aa"
I20260925 10:37:03.201334 19302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be72f30a0b844371a7612cac82be283f" candidate_uuid: "80df0323e95e4a29b2fb86d071083e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653139e7d9d14383b9cec643ac6f8e2e"
I20260925 10:37:03.201490 19563 raft_consensus.cc:3037] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:03.201560 19302 raft_consensus.cc:3037] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:03.203732 19563 raft_consensus.cc:2454] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80df0323e95e4a29b2fb86d071083e7b in term 1.
I20260925 10:37:03.203747 19302 raft_consensus.cc:2454] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80df0323e95e4a29b2fb86d071083e7b in term 1.
I20260925 10:37:03.204106 19366 leader_election.cc:302] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f8ddaaee44047a7933126f112ead5aa, 80df0323e95e4a29b2fb86d071083e7b; no voters:
I20260925 10:37:03.204370 19634 raft_consensus.cc:2781] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:03.204825 19634 raft_consensus.cc:686] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [term 1 LEADER]: Becoming Leader. State: Replica: 80df0323e95e4a29b2fb86d071083e7b, State: Running, Role: LEADER
I20260925 10:37:03.205274 19634 consensus_queue.cc:237] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8ddaaee44047a7933126f112ead5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "80df0323e95e4a29b2fb86d071083e7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } } peers { permanent_uuid: "653139e7d9d14383b9cec643ac6f8e2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39465 } }
I20260925 10:37:03.210026 19160 catalog_manager.cc:5495] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 80df0323e95e4a29b2fb86d071083e7b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "80df0323e95e4a29b2fb86d071083e7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8ddaaee44047a7933126f112ead5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80df0323e95e4a29b2fb86d071083e7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "653139e7d9d14383b9cec643ac6f8e2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39465 } health_report { overall_health: UNKNOWN } } }
W20260925 10:37:03.235736 19609 tablet.cc:2369] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:37:03.239871 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.234062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55930 (local address 127.0.97.2:37327)
0925 10:37:03.234260 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:03.234266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:03.234489 (+ 223us) server_negotiation.cc:407] Connection header received
0925 10:37:03.234739 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:03.234743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:03.234813 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:03.234963 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:03.235338 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.236431 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.237760 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.238154 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.238247 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:03.238798 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:03.239294 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:03.239415 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:03.239538 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 10:37:03.239596 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:37:03.247618 19563 raft_consensus.cc:1261] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa [term 1 FOLLOWER]: Refusing update from remote peer 80df0323e95e4a29b2fb86d071083e7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:37:03.247668 19302 raft_consensus.cc:1261] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e [term 1 FOLLOWER]: Refusing update from remote peer 80df0323e95e4a29b2fb86d071083e7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:37:03.248548 19637 consensus_queue.cc:1035] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8ddaaee44047a7933126f112ead5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:37:03.248848 19634 consensus_queue.cc:1035] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "653139e7d9d14383b9cec643ac6f8e2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:37:03.261562 19642 mvcc.cc:205] Tried to move back new op lower bound from 7333202424812941312 to 7333202424653279232. Current Snapshot: MvccSnapshot[applied={T|T < 7333202424812941312}]
I20260925 10:37:03.271258 19160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:37:03.271516 19160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:37:03.274685 19160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56d212bc5dfa4e0096ebdfc987851818: Sending DeleteTablet for 3 replicas of tablet be72f30a0b844371a7612cac82be283f
I20260925 10:37:03.275661 19543 tablet_service.cc:1514] Processing DeleteTablet for tablet be72f30a0b844371a7612cac82be283f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:37:03 UTC) from {username='slave'} at 127.0.0.1:39092
I20260925 10:37:03.275872 19282 tablet_service.cc:1514] Processing DeleteTablet for tablet be72f30a0b844371a7612cac82be283f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:37:03 UTC) from {username='slave'} at 127.0.0.1:41084
I20260925 10:37:03.275961 19413 tablet_service.cc:1514] Processing DeleteTablet for tablet be72f30a0b844371a7612cac82be283f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:37:03 UTC) from {username='slave'} at 127.0.0.1:55916
I20260925 10:37:03.276520 19649 tablet_replica.cc:330] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa: stopping tablet replica
I20260925 10:37:03.276729 19650 tablet_replica.cc:330] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e: stopping tablet replica
I20260925 10:37:03.276962 19649 raft_consensus.cc:2229] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:37:03.277246 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 19220
I20260925 10:37:03.277223 19650 raft_consensus.cc:2229] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:37:03.277437 19649 raft_consensus.cc:2258] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:37:03.277659 19650 raft_consensus.cc:2258] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:37:03.279937 19650 ts_tablet_manager.cc:1905] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:37:03.284411 19649 ts_tablet_manager.cc:1905] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:37:03.286944 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.278224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.0.97.1:39465)
0925 10:37:03.278435 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:03.278441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:03.278844 (+ 403us) server_negotiation.cc:407] Connection header received
0925 10:37:03.279116 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:03.279122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:03.279213 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:03.279289 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:03.280007 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.281143 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.283086 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.283716 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.283809 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:03.285218 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:03.285229 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:03.285247 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:03.285977 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:03.286039 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:03.286067 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:03.286154 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:03.286445 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 10:37:03.286548 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:37:03.287928 19650 ts_tablet_manager.cc:1918] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:37:03.288110 19342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:03.288154 19650 log.cc:1196] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/be72f30a0b844371a7612cac82be283f
I20260925 10:37:03.297402 19649 ts_tablet_manager.cc:1918] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:37:03.297753 19649 log.cc:1196] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/be72f30a0b844371a7612cac82be283f
I20260925 10:37:03.298229 19649 ts_tablet_manager.cc:1939] T be72f30a0b844371a7612cac82be283f P 7f8ddaaee44047a7933126f112ead5aa: Deleting consensus metadata
I20260925 10:37:03.299860 19145 catalog_manager.cc:4841] TS 7f8ddaaee44047a7933126f112ead5aa (127.0.97.3:45995): tablet be72f30a0b844371a7612cac82be283f (table test-table [id=07b02e0124a04c2697da1a6f0692cab5]) successfully deleted
I20260925 10:37:03.302800 19651 tablet_replica.cc:330] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b: stopping tablet replica
I20260925 10:37:03.303272 19651 raft_consensus.cc:2229] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:37:03.303960 19651 raft_consensus.cc:2258] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:37:03.306015 19651 ts_tablet_manager.cc:1905] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:37:03.313119 19651 ts_tablet_manager.cc:1918] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:37:03.313282 19651 log.cc:1196] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/be72f30a0b844371a7612cac82be283f
I20260925 10:37:03.313882 19651 ts_tablet_manager.cc:1939] T be72f30a0b844371a7612cac82be283f P 80df0323e95e4a29b2fb86d071083e7b: Deleting consensus metadata
I20260925 10:37:03.315611 19148 catalog_manager.cc:4841] TS 80df0323e95e4a29b2fb86d071083e7b (127.0.97.2:37327): tablet be72f30a0b844371a7612cac82be283f (table test-table [id=07b02e0124a04c2697da1a6f0692cab5]) successfully deleted
I20260925 10:37:03.380963 19650 ts_tablet_manager.cc:1939] T be72f30a0b844371a7612cac82be283f P 653139e7d9d14383b9cec643ac6f8e2e: Deleting consensus metadata
I20260925 10:37:03.381014 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 19220
I20260925 10:37:03.398192 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 19351
W20260925 10:37:03.398401 19148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39465 (error 108)
I20260925 10:37:03.401373 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.400860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39465 (local address 127.0.0.1:41104)
0925 10:37:03.401053 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:03.401168 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 10:37:03.401643 19148 catalog_manager.cc:4584] TS 653139e7d9d14383b9cec643ac6f8e2e (127.0.97.1:39465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be72f30a0b844371a7612cac82be283f: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39465: connect: Connection refused (error 111)
I20260925 10:37:03.406265 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.399307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55938 (local address 127.0.97.2:37327)
0925 10:37:03.399483 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:03.399487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:03.399501 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:37:03.399550 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:03.399553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:03.399622 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:03.399734 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:03.400216 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.401256 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.403224 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.403637 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.403728 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:03.404890 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:03.404903 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:03.404925 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:03.405574 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:03.405623 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:03.405641 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:03.405717 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:03.405873 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 10:37:03.405945 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:37:03.407078 19473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:03.455375 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.454859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39465 (local address 127.0.0.1:41116)
0925 10:37:03.455085 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:03.455189 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 10:37:03.456415 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.456031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39465 (local address 127.0.0.1:41124)
0925 10:37:03.456183 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:03.456263 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 10:37:03.515628 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 19351
I20260925 10:37:03.517302 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.516759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39465 (local address 127.0.0.1:41126)
0925 10:37:03.516988 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:03.517099 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 10:37:03.518438 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.517979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39465 (local address 127.0.0.1:41138)
0925 10:37:03.518146 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:03.518250 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 10:37:03.534852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 19481
I20260925 10:37:03.541281 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.535534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39108 (local address 127.0.97.3:45995)
0925 10:37:03.535711 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:03.535718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:03.535736 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:37:03.535921 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:03.535925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:03.536002 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:03.536102 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:03.536491 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.537220 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.538467 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:03.538842 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:03.538969 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:03.540050 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:03.540058 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:03.540072 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:03.540601 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:03.540644 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:03.540662 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:03.540775 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:03.540886 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 10:37:03.540984 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:37:03.541885 19603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:03.599054 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.598535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39465 (local address 127.0.0.1:41154)
0925 10:37:03.598747 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:03.598866 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 10:37:03.600142 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:03.599699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39465 (local address 127.0.0.1:41170)
0925 10:37:03.599919 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:03.599997 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 10:37:03.629106 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 19481
I20260925 10:37:03.645394 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 19130
I20260925 10:37:03.646485 19191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:03.772466 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 19130
2026-09-25T10:37:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2686 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:03 dist-test-slave-dxdd krb5kdc[19660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:03 dist-test-slave-dxdd krb5kdc[19660](info): set up 2 sockets
Sep 25 10:37:03 dist-test-slave-dxdd krb5kdc[19660](info): commencing operation
krb5kdc: starting...
W20260925 10:37:04.858574 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:05 dist-test-slave-dxdd krb5kdc[19660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:37:05.107188 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45659
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:05.419668 19673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:05.419986 19673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:05.420163 19673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:05.427153 19673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:05.427296 19673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:05.427398 19673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:05.427484 19673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:05.427567 19673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:05.436797 19673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:05.438011 19673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:05.439388 19673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:05.450582 19678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:05.450646 19679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:05.450703 19681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:05.451671 19673 server_base.cc:1034] running on GCE node
I20260925 10:37:05.452394 19673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:05.453678 19673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:05.454869 19673 hybrid_clock.cc:648] HybridClock initialized: now 1790332625454809 us; error 47 us; skew 500 ppm
I20260925 10:37:05.455202 19673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:05 dist-test-slave-dxdd krb5kdc[19660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:05.464995 19673 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:37:05.466490 19673 webserver.cc:460] Webserver started at http://127.0.97.62:36217/ using document root <none> and password file <none>
I20260925 10:37:05.467168 19673 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:05.467310 19673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:05.467610 19673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:05.470450 19673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "7185483b03af42b4b88bc9131dae8857"
format_stamp: "Formatted at 2026-09-25 10:37:05 on dist-test-slave-dxdd"
server_key: "a81300cc6048feeb5b468dbf36dab350"
server_key_iv: "cb55ce3be3ae4fc266e24e5aa7ebe459"
server_key_version: "encryptionkey@0"
I20260925 10:37:05.471199 19673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "7185483b03af42b4b88bc9131dae8857"
format_stamp: "Formatted at 2026-09-25 10:37:05 on dist-test-slave-dxdd"
server_key: "a81300cc6048feeb5b468dbf36dab350"
server_key_iv: "cb55ce3be3ae4fc266e24e5aa7ebe459"
server_key_version: "encryptionkey@0"
I20260925 10:37:05.475994 19673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:37:05.479175 19688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:05.480397 19673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 10:37:05.480573 19673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "7185483b03af42b4b88bc9131dae8857"
format_stamp: "Formatted at 2026-09-25 10:37:05 on dist-test-slave-dxdd"
server_key: "a81300cc6048feeb5b468dbf36dab350"
server_key_iv: "cb55ce3be3ae4fc266e24e5aa7ebe459"
server_key_version: "encryptionkey@0"
I20260925 10:37:05.480764 19673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:05.497179 19673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:05.501507 19673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:05.501775 19673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:05.519395 19673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45659
I20260925 10:37:05.519428 19739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45659 every 8 connection(s)
I20260925 10:37:05.520730 19673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:05.524267 19740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:05.527127 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 19673
I20260925 10:37:05.527312 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:05.527735 388 external_mini_cluster.cc:1428] Setting key 82392ae64a62d4c1716ca7951cf0997a
I20260925 10:37:05.535889 19740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857: Bootstrap starting.
I20260925 10:37:05.539538 19740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:05.540802 19740 log.cc:824] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:37:05 dist-test-slave-dxdd krb5kdc[19660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:37:05.543782 19740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857: No bootstrap required, opened a new log
I20260925 10:37:05.551219 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:05.529456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.0.97.62:45659)
0925 10:37:05.530191 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:05.530202 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:05.530242 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:37:05.531929 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:05.531950 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:05.532339 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:05.532656 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:05.534475 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:05.535722 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:05.537498 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:05.538137 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:05.538259 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:05.542959 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:05.542971 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:05.542989 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:05.543029 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:05.547016 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:05.547641 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:05.547648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:05.547663 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:05.547863 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:05.548214 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:05.548219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:05.548222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:05.548856 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:05.549157 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:05.549421 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 10:37:05.549890 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":343,"threads_started":1}
I20260925 10:37:05.554199 19740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7185483b03af42b4b88bc9131dae8857" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45659 } }
I20260925 10:37:05.554558 19740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:05.554630 19740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7185483b03af42b4b88bc9131dae8857, State: Initialized, Role: FOLLOWER
I20260925 10:37:05.555071 19740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7185483b03af42b4b88bc9131dae8857" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45659 } }
I20260925 10:37:05.555279 19740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:05.555374 19740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:05.555490 19740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:05.557464 19740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7185483b03af42b4b88bc9131dae8857" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45659 } }
I20260925 10:37:05.557862 19740 leader_election.cc:302] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7185483b03af42b4b88bc9131dae8857; no voters:
I20260925 10:37:05.558399 19740 leader_election.cc:288] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:05.558526 19745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:05.559186 19745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [term 1 LEADER]: Becoming Leader. State: Replica: 7185483b03af42b4b88bc9131dae8857, State: Running, Role: LEADER
I20260925 10:37:05.559734 19745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7185483b03af42b4b88bc9131dae8857" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45659 } }
I20260925 10:37:05.560508 19740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:05.562855 19747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7185483b03af42b4b88bc9131dae8857. Latest consensus state: current_term: 1 leader_uuid: "7185483b03af42b4b88bc9131dae8857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7185483b03af42b4b88bc9131dae8857" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45659 } } }
I20260925 10:37:05.563277 19747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:05.566810 19746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7185483b03af42b4b88bc9131dae8857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7185483b03af42b4b88bc9131dae8857" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45659 } } }
I20260925 10:37:05.567196 19746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:05.567620 19754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:05.573143 19754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:05.581879 19754 catalog_manager.cc:1347] Generated new cluster ID: 0fb35c3602fd4390a383d3a253a09a69
I20260925 10:37:05.581995 19754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:05.596662 19754 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:05.597968 19754 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:05.606105 19754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857: Generated new TSK 0
I20260925 10:37:05.607084 19754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260925 10:37:05.677326 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45659
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:06.005717 19766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:06.005995 19766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:06.006088 19766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:06.013393 19766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:06.013554 19766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:06.013688 19766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:37:06.023433 19766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:06.024653 19766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:06.025849 19766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:06.038520 19774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:06.038489 19772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:06.038606 19771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:06.039660 19766 server_base.cc:1034] running on GCE node
I20260925 10:37:06.040190 19766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:06.041059 19766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:06.042271 19766 hybrid_clock.cc:648] HybridClock initialized: now 1790332626042216 us; error 55 us; skew 500 ppm
I20260925 10:37:06.042726 19766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:06 dist-test-slave-dxdd krb5kdc[19660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:06.045840 19766 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260925 10:37:06.047276 19766 webserver.cc:460] Webserver started at http://127.0.97.1:44771/ using document root <none> and password file <none>
I20260925 10:37:06.047925 19766 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:06.048051 19766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:06.048350 19766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:06.051174 19766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "80caa6f513644a8981ebce27b17db22d"
format_stamp: "Formatted at 2026-09-25 10:37:06 on dist-test-slave-dxdd"
server_key: "69fb265bbc388c34e98b4566654f9067"
server_key_iv: "b43813564cc9dde6a0f7fb61c64a4d52"
server_key_version: "encryptionkey@0"
I20260925 10:37:06.051944 19766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "80caa6f513644a8981ebce27b17db22d"
format_stamp: "Formatted at 2026-09-25 10:37:06 on dist-test-slave-dxdd"
server_key: "69fb265bbc388c34e98b4566654f9067"
server_key_iv: "b43813564cc9dde6a0f7fb61c64a4d52"
server_key_version: "encryptionkey@0"
I20260925 10:37:06.056978 19766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 10:37:06.060315 19781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:06.061707 19766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:37:06.061903 19766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "80caa6f513644a8981ebce27b17db22d"
format_stamp: "Formatted at 2026-09-25 10:37:06 on dist-test-slave-dxdd"
server_key: "69fb265bbc388c34e98b4566654f9067"
server_key_iv: "b43813564cc9dde6a0f7fb61c64a4d52"
server_key_version: "encryptionkey@0"
I20260925 10:37:06.062103 19766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:06.080488 19766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:06.084694 19766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:06.084980 19766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:06.085951 19766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:37:06.087296 19766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:37:06.087365 19766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:06.087531 19766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:37:06.087586 19766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:06.120121 19766 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35217
I20260925 10:37:06.120170 19893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35217 every 8 connection(s)
I20260925 10:37:06.121430 19766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:37:06.124233 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 19766
I20260925 10:37:06.124410 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:37:06.124811 388 external_mini_cluster.cc:1428] Setting key 43d10c719612a61ec3a16f4c4f65ba4d
Sep 25 10:37:06 dist-test-slave-dxdd krb5kdc[19660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:37:06.141562 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:06.123923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53205 (local address 127.0.97.62:45659)
0925 10:37:06.124435 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:06.124442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:06.125570 (+ 1128us) server_negotiation.cc:407] Connection header received
0925 10:37:06.127469 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:06.127477 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:06.127563 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:06.127687 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:06.129079 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:06.130195 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:06.131737 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:06.132463 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:06.132613 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:06.135931 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:06.135936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:06.135939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:06.135969 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:06.138575 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:06.139295 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:06.139303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:06.139311 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:06.139421 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:06.139854 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:06.139859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:06.139864 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:06.140113 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:06.140264 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:06.140998 (+ 734us) server_negotiation.cc:299] Negotiation successful
0925 10:37:06.141214 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":241,"threads_started":1}
I20260925 10:37:06.142582 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:06.124600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45659 (local address 127.0.97.1:53205)
0925 10:37:06.125407 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:06.125451 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:06.127209 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:06.127876 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:06.127883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:06.128372 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:06.128912 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:06.128929 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:06.130397 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:06.130401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:06.131585 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:06.131607 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:06.132668 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:06.132674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:06.132861 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:06.133518 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:06.133543 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:06.135717 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:06.138715 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:06.138723 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:06.138739 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:06.139168 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:06.139538 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:06.139542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:06.139546 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:06.139728 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:06.140239 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:06.140247 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:06.140566 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 10:37:06.140870 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 10:37:06.141395 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":339,"threads_started":1}
I20260925 10:37:06.144176 19894 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45659
I20260925 10:37:06.144465 19894 heartbeater.cc:464] Registering TS with master...
I20260925 10:37:06.145246 19894 heartbeater.cc:511] Master 127.0.97.62:45659 requested a full tablet report, sending...
I20260925 10:37:06.147267 19705 ts_manager.cc:194] Registered new tserver with Master: 80caa6f513644a8981ebce27b17db22d (127.0.97.1:35217)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260925 10:37:06.148669 19705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260925 10:37:06.177662 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45659
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:06.489395 19900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:06.489646 19900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:06.489734 19900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:06.496635 19900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:06.496726 19900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:06.496832 19900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:37:06.505981 19900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:06.507074 19900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:06.508193 19900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:06.524421 19905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:06.524793 19906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:06.525379 19900 server_base.cc:1034] running on GCE node
W20260925 10:37:06.526525 19908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:06.527209 19900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:06.528192 19900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:06.529404 19900 hybrid_clock.cc:648] HybridClock initialized: now 1790332626529348 us; error 46 us; skew 500 ppm
I20260925 10:37:06.529721 19900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:06 dist-test-slave-dxdd krb5kdc[19660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:06.532917 19900 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260925 10:37:06.534381 19900 webserver.cc:460] Webserver started at http://127.0.97.2:44247/ using document root <none> and password file <none>
I20260925 10:37:06.535003 19900 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:06.535144 19900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:06.535450 19900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:06.538128 19900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "6888e94a6a1d41469a72e2ac90e66ce5"
format_stamp: "Formatted at 2026-09-25 10:37:06 on dist-test-slave-dxdd"
server_key: "22f41c6dccc1991704037ace867c0f6b"
server_key_iv: "4a8af4efc5dd1ca6f74cdb5da081d3c7"
server_key_version: "encryptionkey@0"
I20260925 10:37:06.538949 19900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6888e94a6a1d41469a72e2ac90e66ce5"
format_stamp: "Formatted at 2026-09-25 10:37:06 on dist-test-slave-dxdd"
server_key: "22f41c6dccc1991704037ace867c0f6b"
server_key_iv: "4a8af4efc5dd1ca6f74cdb5da081d3c7"
server_key_version: "encryptionkey@0"
I20260925 10:37:06.543637 19900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:37:06.546890 19915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:06.548120 19900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:37:06.548280 19900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "6888e94a6a1d41469a72e2ac90e66ce5"
format_stamp: "Formatted at 2026-09-25 10:37:06 on dist-test-slave-dxdd"
server_key: "22f41c6dccc1991704037ace867c0f6b"
server_key_iv: "4a8af4efc5dd1ca6f74cdb5da081d3c7"
server_key_version: "encryptionkey@0"
I20260925 10:37:06.548461 19900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:06.561447 19900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:06.564967 19900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:06.565269 19900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:06.566180 19900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:37:06.567448 19900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:37:06.567546 19900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:06.567688 19900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:37:06.567761 19900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:06.599771 19900 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38199
I20260925 10:37:06.599852 20027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38199 every 8 connection(s)
I20260925 10:37:06.601086 19900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:37:06.610050 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 19900
I20260925 10:37:06.610241 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:37:06.610801 388 external_mini_cluster.cc:1428] Setting key 08de3647e6ebb33d2e2950e4ac562541
Sep 25 10:37:06 dist-test-slave-dxdd krb5kdc[19660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:37:06.620939 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:06.603506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33169 (local address 127.0.97.62:45659)
0925 10:37:06.603715 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:06.603721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:06.604772 (+ 1051us) server_negotiation.cc:407] Connection header received
0925 10:37:06.606658 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:06.606662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:06.606727 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:06.606815 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:06.608268 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:06.608968 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:06.610501 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:06.611114 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:06.611247 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:06.615371 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:06.615376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:06.615379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:06.615409 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:06.617785 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:06.618755 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:06.618759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:06.618761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:06.618874 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:06.619339 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:06.619342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:06.619343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:06.619568 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:06.619671 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:06.620532 (+ 861us) server_negotiation.cc:299] Negotiation successful
0925 10:37:06.620688 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:37:06.622063 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:06.603930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45659 (local address 127.0.97.2:33169)
0925 10:37:06.604615 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:06.604662 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:06.606378 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:06.607000 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:06.607008 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:06.607515 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:06.608044 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:06.608061 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:06.609083 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:06.609086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:06.610359 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:06.610375 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:06.611306 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:06.611311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:06.611603 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:06.612294 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:06.612320 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:06.615168 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:06.617925 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:06.617936 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:06.617967 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:06.618645 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:06.618979 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:06.618986 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:06.618993 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:06.619236 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:06.619689 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:06.619700 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:06.620175 (+ 475us) client_negotiation.cc:769] Sending connection context
0925 10:37:06.620456 (+ 281us) client_negotiation.cc:240] Negotiation successful
0925 10:37:06.621020 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":279,"threads_started":1}
I20260925 10:37:06.623445 20028 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45659
I20260925 10:37:06.623749 20028 heartbeater.cc:464] Registering TS with master...
I20260925 10:37:06.624437 20028 heartbeater.cc:511] Master 127.0.97.62:45659 requested a full tablet report, sending...
I20260925 10:37:06.625830 19705 ts_manager.cc:194] Registered new tserver with Master: 6888e94a6a1d41469a72e2ac90e66ce5 (127.0.97.2:38199)
I20260925 10:37:06.626685 19705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33169
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260925 10:37:06.663576 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45659
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:06.974027 20033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:06.974395 20033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:06.974558 20033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:06.981485 20033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:06.981581 20033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:06.981689 20033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:37:06.991019 20033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:06.992228 20033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:06.993459 20033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:07.006217 20038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:07.006271 20039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:07.006139 20041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:07.007077 20033 server_base.cc:1034] running on GCE node
I20260925 10:37:07.007661 20033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:07.008544 20033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:07.009753 20033 hybrid_clock.cc:648] HybridClock initialized: now 1790332627009719 us; error 47 us; skew 500 ppm
I20260925 10:37:07.010052 20033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:07 dist-test-slave-dxdd krb5kdc[19660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:07.012907 20033 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260925 10:37:07.014282 20033 webserver.cc:460] Webserver started at http://127.0.97.3:44715/ using document root <none> and password file <none>
I20260925 10:37:07.015033 20033 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:07.015168 20033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:07.015455 20033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:07.018172 20033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "267effd249bc456993ce9d995f54edfd"
format_stamp: "Formatted at 2026-09-25 10:37:07 on dist-test-slave-dxdd"
server_key: "e63f56fb433330501625eece0b07ed63"
server_key_iv: "7e504810a2d9d8cbc643c02d992c620e"
server_key_version: "encryptionkey@0"
I20260925 10:37:07.018983 20033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "267effd249bc456993ce9d995f54edfd"
format_stamp: "Formatted at 2026-09-25 10:37:07 on dist-test-slave-dxdd"
server_key: "e63f56fb433330501625eece0b07ed63"
server_key_iv: "7e504810a2d9d8cbc643c02d992c620e"
server_key_version: "encryptionkey@0"
I20260925 10:37:07.023723 20033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:37:07.026932 20048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:07.028174 20033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:37:07.028342 20033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "267effd249bc456993ce9d995f54edfd"
format_stamp: "Formatted at 2026-09-25 10:37:07 on dist-test-slave-dxdd"
server_key: "e63f56fb433330501625eece0b07ed63"
server_key_iv: "7e504810a2d9d8cbc643c02d992c620e"
server_key_version: "encryptionkey@0"
I20260925 10:37:07.028533 20033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:07.059345 20033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:07.063431 20033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:07.063714 20033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:07.064661 20033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:37:07.065985 20033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:37:07.066092 20033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:07.066234 20033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:37:07.066306 20033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:07.098745 20033 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41327
I20260925 10:37:07.098791 20160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41327 every 8 connection(s)
I20260925 10:37:07.100015 20033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:37:07.105400 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 20033
I20260925 10:37:07.105607 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:37:07.106137 388 external_mini_cluster.cc:1428] Setting key cc157cd169191a7a3c0fc4e4212dc749
Sep 25 10:37:07 dist-test-slave-dxdd krb5kdc[19660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:37:07.119254 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.102492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54673 (local address 127.0.97.62:45659)
0925 10:37:07.102686 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:07.102690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:07.103837 (+ 1147us) server_negotiation.cc:407] Connection header received
0925 10:37:07.105681 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:07.105687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:07.105777 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:07.105884 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:07.107580 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.108511 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.110082 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.110523 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.110615 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.113820 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:07.113825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:07.113828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:07.113859 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:07.116656 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:07.117370 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:07.117374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:07.117376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:07.117490 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:07.117833 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:07.117836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:07.117838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:07.118086 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:07.118192 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:07.118853 (+ 661us) server_negotiation.cc:299] Negotiation successful
0925 10:37:07.119019 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 10:37:07.120481 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.103031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45659 (local address 127.0.97.3:54673)
0925 10:37:07.103689 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:07.103724 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:07.105365 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:07.106065 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:07.106074 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:07.106775 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:07.107416 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:07.107436 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.108705 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.108709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:07.109943 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:07.109955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.110694 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.110698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:07.110884 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.111554 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:07.111575 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:07.113653 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:07.116848 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:07.116855 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:07.116865 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:07.117275 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:07.117581 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:07.117584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:07.117586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:07.117745 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:07.118187 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:07.118192 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:07.118546 (+ 354us) client_negotiation.cc:769] Sending connection context
0925 10:37:07.118760 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 10:37:07.119190 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":281,"threads_started":1}
I20260925 10:37:07.121721 20161 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45659
I20260925 10:37:07.122017 20161 heartbeater.cc:464] Registering TS with master...
I20260925 10:37:07.122743 20161 heartbeater.cc:511] Master 127.0.97.62:45659 requested a full tablet report, sending...
I20260925 10:37:07.124151 19705 ts_manager.cc:194] Registered new tserver with Master: 267effd249bc456993ce9d995f54edfd (127.0.97.3:41327)
I20260925 10:37:07.124920 19705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54673
I20260925 10:37:07.133502 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:37:07.148860 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.137298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.0.97.62:45659)
0925 10:37:07.137513 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:07.137521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:07.137892 (+ 371us) server_negotiation.cc:407] Connection header received
0925 10:37:07.138022 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:07.138026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:07.138093 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:07.138171 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:07.139536 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.140223 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.141421 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.141797 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.141883 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.144079 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:07.144084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:07.144087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:07.144116 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:07.146698 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:07.147347 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:07.147351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:07.147353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:07.147462 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:07.147740 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:07.147743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:07.147745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:07.148016 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:07.148191 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:07.148418 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 10:37:07.148574 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:37:07.151785 19894 heartbeater.cc:503] Master 127.0.97.62:45659 was elected leader, sending a full tablet report...
I20260925 10:37:07.152559 19705 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:37:07.159521 19705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:37:07.180907 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.171115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36000 (local address 127.0.97.1:35217)
0925 10:37:07.172739 (+ 1624us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:07.172751 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:07.172787 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:37:07.172904 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:07.172936 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:07.173268 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:07.173511 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:07.174239 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.175899 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.178716 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.179648 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.179794 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.180060 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:07.180377 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 10:37:07.180551 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1408,"thread_start_us":1208,"threads_started":1}
I20260925 10:37:07.180956 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.170765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35217 (local address 127.0.0.1:36000)
0925 10:37:07.171751 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:07.171812 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:07.172002 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:07.173543 (+ 1541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:07.173551 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:07.173585 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:07.174021 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:07.174036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.176035 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.176040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:07.178528 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:07.178542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.179837 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.179842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:07.180122 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.180181 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 10:37:07.180311 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 10:37:07.180513 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":79,"thread_start_us":641,"threads_started":1}
I20260925 10:37:07.182835 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.172234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38199 (local address 127.0.0.1:45642)
0925 10:37:07.174299 (+ 2065us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:07.174338 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:07.174452 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:07.175346 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:07.175350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:07.175374 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:07.175910 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:07.175920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.177796 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.177801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:07.180066 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:07.180080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.182118 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.182124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:07.182352 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.182387 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:37:07.182481 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 10:37:07.182567 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1912,"mutex_wait_us":55,"thread_start_us":1779,"threads_started":1}
I20260925 10:37:07.182981 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.172500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.0.97.2:38199)
0925 10:37:07.173017 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:07.173042 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:07.174596 (+ 1554us) server_negotiation.cc:407] Connection header received
0925 10:37:07.174696 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:07.174705 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:07.174928 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:07.175136 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:07.176111 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.177648 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.180217 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.181677 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.181792 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.181960 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:07.182548 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 10:37:07.182710 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":273,"threads_started":1}
I20260925 10:37:07.184774 19829 tablet_service.cc:1467] Processing CreateTablet for tablet 790bb486b9a44a9eb9bd7e5f3eeee274 (DEFAULT_TABLE table=test-table [id=6a66bad3452846e79e223da43d1eb9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:37:07.185746 19963 tablet_service.cc:1467] Processing CreateTablet for tablet 790bb486b9a44a9eb9bd7e5f3eeee274 (DEFAULT_TABLE table=test-table [id=6a66bad3452846e79e223da43d1eb9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:37:07.186172 19829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 790bb486b9a44a9eb9bd7e5f3eeee274. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:07.186964 19963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 790bb486b9a44a9eb9bd7e5f3eeee274. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:07.187013 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.175387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41327 (local address 127.0.0.1:52262)
0925 10:37:07.175978 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:07.176006 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:07.176127 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:07.176621 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:07.176626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:07.176649 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:07.177045 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:07.177054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.178870 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.178875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:07.180856 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:07.180874 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.186187 (+ 5313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.186196 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:07.186412 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.186444 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:37:07.186555 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 10:37:07.186673 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":1,"thread_start_us":305,"threads_started":1}
I20260925 10:37:07.187150 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.174724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.0.97.3:41327)
0925 10:37:07.175417 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:07.175425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:07.176096 (+ 671us) server_negotiation.cc:407] Connection header received
0925 10:37:07.176290 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:07.176297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:07.176447 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:07.176632 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:07.177220 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.178727 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.181078 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.185710 (+ 4632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.185814 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.186011 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:07.186629 (+ 618us) server_negotiation.cc:299] Negotiation successful
0925 10:37:07.186788 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":377,"threads_started":1}
I20260925 10:37:07.190466 20096 tablet_service.cc:1467] Processing CreateTablet for tablet 790bb486b9a44a9eb9bd7e5f3eeee274 (DEFAULT_TABLE table=test-table [id=6a66bad3452846e79e223da43d1eb9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:37:07.191764 20096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 790bb486b9a44a9eb9bd7e5f3eeee274. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:07.197348 20179 tablet_bootstrap.cc:492] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d: Bootstrap starting.
I20260925 10:37:07.202420 20179 tablet_bootstrap.cc:654] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:07.203724 20179 log.cc:824] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:07.205433 20180 tablet_bootstrap.cc:492] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5: Bootstrap starting.
I20260925 10:37:07.208097 20182 tablet_bootstrap.cc:492] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd: Bootstrap starting.
I20260925 10:37:07.208858 20179 tablet_bootstrap.cc:492] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d: No bootstrap required, opened a new log
I20260925 10:37:07.208873 20180 tablet_bootstrap.cc:654] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:07.209779 20179 ts_tablet_manager.cc:1397] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260925 10:37:07.210080 20180 log.cc:824] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:07.212855 20182 tablet_bootstrap.cc:654] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:07.213990 20182 log.cc:824] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:07.214994 20180 tablet_bootstrap.cc:492] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5: No bootstrap required, opened a new log
I20260925 10:37:07.215261 20180 ts_tablet_manager.cc:1397] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260925 10:37:07.216939 20182 tablet_bootstrap.cc:492] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd: No bootstrap required, opened a new log
I20260925 10:37:07.217172 20182 ts_tablet_manager.cc:1397] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 10:37:07.217866 20179 raft_consensus.cc:348] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80caa6f513644a8981ebce27b17db22d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35217 } } peers { permanent_uuid: "6888e94a6a1d41469a72e2ac90e66ce5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38199 } } peers { permanent_uuid: "267effd249bc456993ce9d995f54edfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41327 } }
I20260925 10:37:07.218204 20179 raft_consensus.cc:374] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:07.218322 20179 raft_consensus.cc:729] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80caa6f513644a8981ebce27b17db22d, State: Initialized, Role: FOLLOWER
I20260925 10:37:07.218911 20179 consensus_queue.cc:260] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80caa6f513644a8981ebce27b17db22d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35217 } } peers { permanent_uuid: "6888e94a6a1d41469a72e2ac90e66ce5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38199 } } peers { permanent_uuid: "267effd249bc456993ce9d995f54edfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41327 } }
I20260925 10:37:07.221807 20179 ts_tablet_manager.cc:1428] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260925 10:37:07.225569 20180 raft_consensus.cc:348] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80caa6f513644a8981ebce27b17db22d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35217 } } peers { permanent_uuid: "6888e94a6a1d41469a72e2ac90e66ce5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38199 } } peers { permanent_uuid: "267effd249bc456993ce9d995f54edfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41327 } }
I20260925 10:37:07.225997 20180 raft_consensus.cc:374] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:07.226100 20180 raft_consensus.cc:729] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6888e94a6a1d41469a72e2ac90e66ce5, State: Initialized, Role: FOLLOWER
I20260925 10:37:07.226711 20180 consensus_queue.cc:260] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80caa6f513644a8981ebce27b17db22d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35217 } } peers { permanent_uuid: "6888e94a6a1d41469a72e2ac90e66ce5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38199 } } peers { permanent_uuid: "267effd249bc456993ce9d995f54edfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41327 } }
I20260925 10:37:07.227682 20028 heartbeater.cc:503] Master 127.0.97.62:45659 was elected leader, sending a full tablet report...
I20260925 10:37:07.227583 20182 raft_consensus.cc:348] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80caa6f513644a8981ebce27b17db22d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35217 } } peers { permanent_uuid: "6888e94a6a1d41469a72e2ac90e66ce5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38199 } } peers { permanent_uuid: "267effd249bc456993ce9d995f54edfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41327 } }
I20260925 10:37:07.227979 20182 raft_consensus.cc:374] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:07.228081 20182 raft_consensus.cc:729] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 267effd249bc456993ce9d995f54edfd, State: Initialized, Role: FOLLOWER
I20260925 10:37:07.228422 20180 ts_tablet_manager.cc:1428] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 10:37:07.228665 20182 consensus_queue.cc:260] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80caa6f513644a8981ebce27b17db22d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35217 } } peers { permanent_uuid: "6888e94a6a1d41469a72e2ac90e66ce5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38199 } } peers { permanent_uuid: "267effd249bc456993ce9d995f54edfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41327 } }
I20260925 10:37:07.229811 20161 heartbeater.cc:503] Master 127.0.97.62:45659 was elected leader, sending a full tablet report...
I20260925 10:37:07.230403 20182 ts_tablet_manager.cc:1428] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd: Time spent starting tablet: real 0.013s user 0.005s sys 0.010s
I20260925 10:37:07.326216 20186 raft_consensus.cc:482] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:37:07.326524 20186 raft_consensus.cc:504] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80caa6f513644a8981ebce27b17db22d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35217 } } peers { permanent_uuid: "6888e94a6a1d41469a72e2ac90e66ce5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38199 } } peers { permanent_uuid: "267effd249bc456993ce9d995f54edfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41327 } }
I20260925 10:37:07.327978 20186 leader_election.cc:288] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80caa6f513644a8981ebce27b17db22d (127.0.97.1:35217), 267effd249bc456993ce9d995f54edfd (127.0.97.3:41327)
I20260925 10:37:07.335246 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.329463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50277 (local address 127.0.97.3:41327)
0925 10:37:07.329672 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:07.329679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:07.329931 (+ 252us) server_negotiation.cc:407] Connection header received
0925 10:37:07.330057 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:07.330065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:07.330144 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:07.330241 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:07.330847 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.331697 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.333470 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.334208 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.334356 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.334480 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:07.334835 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 10:37:07.334986 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 10:37:07.335297 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.328484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51287 (local address 127.0.97.1:35217)
0925 10:37:07.328667 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:07.328672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:07.329216 (+ 544us) server_negotiation.cc:407] Connection header received
0925 10:37:07.329417 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:07.329423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:07.329499 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:07.329597 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:07.330328 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.331241 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.333472 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.334245 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.334445 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.334555 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:07.334875 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 10:37:07.335000 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:37:07.335443 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.329228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41327 (local address 127.0.97.2:50277)
0925 10:37:07.329816 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:07.329828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:07.329948 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:07.330285 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:07.330289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:07.330312 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:07.330687 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:07.330700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.331862 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.331868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:07.333327 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:07.333338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.334460 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.334467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:07.334657 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.334700 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 10:37:07.334774 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:37:07.334960 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":2,"thread_start_us":315,"threads_started":1}
I20260925 10:37:07.335621 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.328362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35217 (local address 127.0.97.2:51287)
0925 10:37:07.329110 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:07.329131 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:07.329265 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:07.329762 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:07.329766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:07.329805 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:07.330164 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:07.330178 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.331377 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.331382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:07.333325 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:07.333337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.334530 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.334539 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:07.334667 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.334720 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 10:37:07.334835 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 10:37:07.334958 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":61,"thread_start_us":404,"threads_started":1}
I20260925 10:37:07.336514 20116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "790bb486b9a44a9eb9bd7e5f3eeee274" candidate_uuid: "6888e94a6a1d41469a72e2ac90e66ce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "267effd249bc456993ce9d995f54edfd" is_pre_election: true
I20260925 10:37:07.336539 19849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "790bb486b9a44a9eb9bd7e5f3eeee274" candidate_uuid: "6888e94a6a1d41469a72e2ac90e66ce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80caa6f513644a8981ebce27b17db22d" is_pre_election: true
I20260925 10:37:07.337021 20116 raft_consensus.cc:2454] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6888e94a6a1d41469a72e2ac90e66ce5 in term 0.
I20260925 10:37:07.337020 19849 raft_consensus.cc:2454] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6888e94a6a1d41469a72e2ac90e66ce5 in term 0.
I20260925 10:37:07.337649 19916 leader_election.cc:302] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 267effd249bc456993ce9d995f54edfd, 6888e94a6a1d41469a72e2ac90e66ce5; no voters:
I20260925 10:37:07.338019 20186 raft_consensus.cc:2781] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:37:07.338171 20186 raft_consensus.cc:482] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:37:07.338281 20186 raft_consensus.cc:3037] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:07.340502 20186 raft_consensus.cc:504] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80caa6f513644a8981ebce27b17db22d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35217 } } peers { permanent_uuid: "6888e94a6a1d41469a72e2ac90e66ce5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38199 } } peers { permanent_uuid: "267effd249bc456993ce9d995f54edfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41327 } }
I20260925 10:37:07.341085 20186 leader_election.cc:288] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [CANDIDATE]: Term 1 election: Requested vote from peers 80caa6f513644a8981ebce27b17db22d (127.0.97.1:35217), 267effd249bc456993ce9d995f54edfd (127.0.97.3:41327)
I20260925 10:37:07.341557 19849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "790bb486b9a44a9eb9bd7e5f3eeee274" candidate_uuid: "6888e94a6a1d41469a72e2ac90e66ce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80caa6f513644a8981ebce27b17db22d"
I20260925 10:37:07.341629 20116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "790bb486b9a44a9eb9bd7e5f3eeee274" candidate_uuid: "6888e94a6a1d41469a72e2ac90e66ce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "267effd249bc456993ce9d995f54edfd"
I20260925 10:37:07.341749 19849 raft_consensus.cc:3037] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:07.341853 20116 raft_consensus.cc:3037] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:07.344115 19849 raft_consensus.cc:2454] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6888e94a6a1d41469a72e2ac90e66ce5 in term 1.
I20260925 10:37:07.344224 20116 raft_consensus.cc:2454] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6888e94a6a1d41469a72e2ac90e66ce5 in term 1.
I20260925 10:37:07.344517 19916 leader_election.cc:302] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6888e94a6a1d41469a72e2ac90e66ce5, 80caa6f513644a8981ebce27b17db22d; no voters:
I20260925 10:37:07.344763 20186 raft_consensus.cc:2781] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:07.345170 20186 raft_consensus.cc:686] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [term 1 LEADER]: Becoming Leader. State: Replica: 6888e94a6a1d41469a72e2ac90e66ce5, State: Running, Role: LEADER
I20260925 10:37:07.345631 20186 consensus_queue.cc:237] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80caa6f513644a8981ebce27b17db22d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35217 } } peers { permanent_uuid: "6888e94a6a1d41469a72e2ac90e66ce5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38199 } } peers { permanent_uuid: "267effd249bc456993ce9d995f54edfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41327 } }
I20260925 10:37:07.350279 19704 catalog_manager.cc:5495] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6888e94a6a1d41469a72e2ac90e66ce5 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6888e94a6a1d41469a72e2ac90e66ce5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80caa6f513644a8981ebce27b17db22d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6888e94a6a1d41469a72e2ac90e66ce5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "267effd249bc456993ce9d995f54edfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41327 } health_report { overall_health: UNKNOWN } } }
W20260925 10:37:07.351634 20162 tablet.cc:2369] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:37:07.352939 20029 tablet.cc:2369] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:37:07.371428 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.366350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45650 (local address 127.0.97.2:38199)
0925 10:37:07.366534 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:07.366538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:07.366554 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:37:07.366740 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:07.366744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:07.366813 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:07.366952 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:07.367287 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.367948 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.369346 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.369750 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.369884 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.370438 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:07.370878 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:07.371041 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:07.371091 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 10:37:07.371166 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 10:37:07.373658 19895 tablet.cc:2369] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:37:07.379081 19849 raft_consensus.cc:1261] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d [term 1 FOLLOWER]: Refusing update from remote peer 6888e94a6a1d41469a72e2ac90e66ce5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:37:07.379081 20116 raft_consensus.cc:1261] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd [term 1 FOLLOWER]: Refusing update from remote peer 6888e94a6a1d41469a72e2ac90e66ce5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:37:07.379967 20190 consensus_queue.cc:1035] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "267effd249bc456993ce9d995f54edfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:37:07.380327 20186 consensus_queue.cc:1035] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80caa6f513644a8981ebce27b17db22d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:37:07.392889 20195 mvcc.cc:205] Tried to move back new op lower bound from 7333202441735282688 to 7333202441612046336. Current Snapshot: MvccSnapshot[applied={T|T < 7333202441735282688}]
I20260925 10:37:07.407156 19705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:37:07.407413 19705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:37:07.411087 19705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7185483b03af42b4b88bc9131dae8857: Sending DeleteTablet for 3 replicas of tablet 790bb486b9a44a9eb9bd7e5f3eeee274
I20260925 10:37:07.412228 19829 tablet_service.cc:1514] Processing DeleteTablet for tablet 790bb486b9a44a9eb9bd7e5f3eeee274 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:37:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36000
I20260925 10:37:07.412335 20096 tablet_service.cc:1514] Processing DeleteTablet for tablet 790bb486b9a44a9eb9bd7e5f3eeee274 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:37:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52262
I20260925 10:37:07.412990 19963 tablet_service.cc:1514] Processing DeleteTablet for tablet 790bb486b9a44a9eb9bd7e5f3eeee274 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:37:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45642
I20260925 10:37:07.413493 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 19766
I20260925 10:37:07.423208 20202 tablet_replica.cc:330] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d: stopping tablet replica
I20260925 10:37:07.423861 20202 raft_consensus.cc:2229] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:37:07.424494 20202 raft_consensus.cc:2258] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:37:07.425338 20204 tablet_replica.cc:330] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5: stopping tablet replica
I20260925 10:37:07.425766 20204 raft_consensus.cc:2229] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:37:07.426568 20204 raft_consensus.cc:2258] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:37:07.426872 20202 ts_tablet_manager.cc:1905] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:37:07.427537 20203 tablet_replica.cc:330] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd: stopping tablet replica
I20260925 10:37:07.428776 20204 ts_tablet_manager.cc:1905] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:37:07.429376 20203 raft_consensus.cc:2229] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:37:07.431902 20203 raft_consensus.cc:2258] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:37:07.434684 20202 ts_tablet_manager.cc:1918] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:37:07.434907 20202 log.cc:1196] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-0/wal/wals/790bb486b9a44a9eb9bd7e5f3eeee274
Sep 25 10:37:07 dist-test-slave-dxdd krb5kdc[19660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260925 10:37:07.435504 20202 ts_tablet_manager.cc:1939] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 80caa6f513644a8981ebce27b17db22d: Deleting consensus metadata
I20260925 10:37:07.437366 19689 catalog_manager.cc:4841] TS 80caa6f513644a8981ebce27b17db22d (127.0.97.1:35217): tablet 790bb486b9a44a9eb9bd7e5f3eeee274 (table test-table [id=6a66bad3452846e79e223da43d1eb9fa]) successfully deleted
I20260925 10:37:07.437477 20203 ts_tablet_manager.cc:1905] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:37:07.442176 20204 ts_tablet_manager.cc:1918] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:37:07.442440 20204 log.cc:1196] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/790bb486b9a44a9eb9bd7e5f3eeee274
I20260925 10:37:07.443038 20204 ts_tablet_manager.cc:1939] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 6888e94a6a1d41469a72e2ac90e66ce5: Deleting consensus metadata
I20260925 10:37:07.444167 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.414655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36008 (local address 127.0.97.1:35217)
0925 10:37:07.414840 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:07.414846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:07.424532 (+ 9686us) server_negotiation.cc:407] Connection header received
0925 10:37:07.424939 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:07.424947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:07.425063 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:07.425138 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:07.426971 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.428096 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.429906 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.430607 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.430704 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.436874 (+ 6170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:07.436887 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:07.436902 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:07.436945 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:07.440592 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:07.441515 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:07.441523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:07.441527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:07.441703 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:07.442186 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:07.442190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:07.442194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:07.442661 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:07.442963 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:07.443531 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 10:37:07.443758 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:37:07.444361 20203 ts_tablet_manager.cc:1918] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:37:07.444514 20203 log.cc:1196] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/790bb486b9a44a9eb9bd7e5f3eeee274
I20260925 10:37:07.445075 20203 ts_tablet_manager.cc:1939] T 790bb486b9a44a9eb9bd7e5f3eeee274 P 267effd249bc456993ce9d995f54edfd: Deleting consensus metadata
I20260925 10:37:07.445422 19689 catalog_manager.cc:4841] TS 6888e94a6a1d41469a72e2ac90e66ce5 (127.0.97.2:38199): tablet 790bb486b9a44a9eb9bd7e5f3eeee274 (table test-table [id=6a66bad3452846e79e223da43d1eb9fa]) successfully deleted
I20260925 10:37:07.445577 19889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:07.448487 19689 catalog_manager.cc:4841] TS 267effd249bc456993ce9d995f54edfd (127.0.97.3:41327): tablet 790bb486b9a44a9eb9bd7e5f3eeee274 (table test-table [id=6a66bad3452846e79e223da43d1eb9fa]) successfully deleted
I20260925 10:37:07.535836 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 19766
I20260925 10:37:07.553855 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 19900
Sep 25 10:37:07 dist-test-slave-dxdd krb5kdc[19660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260925 10:37:07.570879 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.554938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.0.97.2:38199)
0925 10:37:07.555109 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:07.555113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:07.555128 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:37:07.555180 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:07.555184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:07.555244 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:07.555325 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:07.556958 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.557763 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.559589 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.560224 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.560348 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.564965 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:07.564977 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:07.564993 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:07.565037 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:07.567945 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:07.568760 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:07.568766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:07.568768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:07.568902 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:07.569272 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:07.569275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:07.569277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:07.569607 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:07.569840 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:07.570130 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 10:37:07.570279 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:37:07.571678 20023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:07.681699 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 19900
I20260925 10:37:07.701359 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 20033
Sep 25 10:37:07 dist-test-slave-dxdd krb5kdc[19660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260925 10:37:07.716080 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:07.702088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.0.97.3:41327)
0925 10:37:07.702267 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:07.702275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:07.702345 (+ 70us) server_negotiation.cc:407] Connection header received
0925 10:37:07.702586 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:07.702592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:07.702660 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:07.702769 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:07.704067 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.704785 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.706054 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:07.706513 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:07.706659 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:07.710847 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:07.710858 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:07.710873 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:07.710904 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:07.713292 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:07.713933 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:07.713941 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:07.713949 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:07.714088 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:07.714600 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:07.714607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:07.714611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:07.714963 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:07.715246 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:07.715554 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 10:37:07.715778 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:37:07.716668 20156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:07.803849 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 20033
I20260925 10:37:07.820238 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 19673
I20260925 10:37:07.821293 19735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:07.928795 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 19673
2026-09-25T10:37:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4157 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T10:37:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:07Z Disabled control of system clock
I20260925 10:37:08.002955 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44833
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:37:08.321308 20213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:08.321648 20213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:08.321750 20213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:08.328755 20213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:08.328853 20213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:08.328900 20213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:08.328958 20213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:08.329000 20213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:08.337679 20213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44833
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:08.338843 20213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:08.340507 20213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:08.351958 20218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:08.352030 20219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:08.351997 20221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:08.353178 20213 server_base.cc:1034] running on GCE node
I20260925 10:37:08.353964 20213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:08.355391 20213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:08.356580 20213 hybrid_clock.cc:648] HybridClock initialized: now 1790332628356525 us; error 53 us; skew 500 ppm
I20260925 10:37:08.356915 20213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:37:08.359086 20213 webserver.cc:460] Webserver started at http://127.0.97.62:35351/ using document root <none> and password file <none>
I20260925 10:37:08.359735 20213 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:08.359843 20213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:08.360158 20213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:08.363070 20213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "2f2312e3c46744fd84f08dbe02895afc"
format_stamp: "Formatted at 2026-09-25 10:37:08 on dist-test-slave-dxdd"
server_key: "37c6cb4d5875f819a74afb22f15cff15"
server_key_iv: "00629bff653e8d8224b67f501e21d731"
server_key_version: "encryptionkey@0"
I20260925 10:37:08.363850 20213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "2f2312e3c46744fd84f08dbe02895afc"
format_stamp: "Formatted at 2026-09-25 10:37:08 on dist-test-slave-dxdd"
server_key: "37c6cb4d5875f819a74afb22f15cff15"
server_key_iv: "00629bff653e8d8224b67f501e21d731"
server_key_version: "encryptionkey@0"
I20260925 10:37:08.368664 20213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 10:37:08.371841 20227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:08.373045 20213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:37:08.373227 20213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "2f2312e3c46744fd84f08dbe02895afc"
format_stamp: "Formatted at 2026-09-25 10:37:08 on dist-test-slave-dxdd"
server_key: "37c6cb4d5875f819a74afb22f15cff15"
server_key_iv: "00629bff653e8d8224b67f501e21d731"
server_key_version: "encryptionkey@0"
I20260925 10:37:08.373430 20213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:08.397195 20213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:08.397994 20213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:08.398276 20213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:08.415706 20213 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44833
I20260925 10:37:08.415728 20278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44833 every 8 connection(s)
I20260925 10:37:08.416994 20213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:08.420463 20279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:08.423355 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 20213
I20260925 10:37:08.423537 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:08.424055 388 external_mini_cluster.cc:1428] Setting key 1dece167725fd2338d60d108db76d53f
I20260925 10:37:08.431648 20279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc: Bootstrap starting.
I20260925 10:37:08.434496 20279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:08.435536 20279 log.cc:824] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:08.437152 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:08.425858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.0.97.62:44833)
0925 10:37:08.426860 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:08.426870 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:08.426909 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:37:08.428497 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:08.428517 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:08.428946 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:08.429323 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:08.429874 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:08.431160 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:08.432552 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:08.433227 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:08.433444 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:08.434615 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:08.434625 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:08.434642 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:08.435200 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:08.435294 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:08.435314 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:08.435471 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:08.435660 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 10:37:08.435964 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":440,"threads_started":1}
I20260925 10:37:08.438643 20279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc: No bootstrap required, opened a new log
I20260925 10:37:08.445896 20279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2312e3c46744fd84f08dbe02895afc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44833 } }
I20260925 10:37:08.446224 20279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:08.446338 20279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f2312e3c46744fd84f08dbe02895afc, State: Initialized, Role: FOLLOWER
I20260925 10:37:08.446869 20279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2312e3c46744fd84f08dbe02895afc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44833 } }
I20260925 10:37:08.447103 20279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:08.447229 20279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:08.447396 20279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:08.449384 20279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2312e3c46744fd84f08dbe02895afc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44833 } }
I20260925 10:37:08.449875 20279 leader_election.cc:302] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f2312e3c46744fd84f08dbe02895afc; no voters:
I20260925 10:37:08.450479 20279 leader_election.cc:288] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:08.450613 20284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:08.451993 20284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [term 1 LEADER]: Becoming Leader. State: Replica: 2f2312e3c46744fd84f08dbe02895afc, State: Running, Role: LEADER
I20260925 10:37:08.452486 20279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:08.452582 20284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2312e3c46744fd84f08dbe02895afc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44833 } }
I20260925 10:37:08.455128 20286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f2312e3c46744fd84f08dbe02895afc. Latest consensus state: current_term: 1 leader_uuid: "2f2312e3c46744fd84f08dbe02895afc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2312e3c46744fd84f08dbe02895afc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44833 } } }
I20260925 10:37:08.455498 20286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:08.455979 20285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f2312e3c46744fd84f08dbe02895afc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2312e3c46744fd84f08dbe02895afc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44833 } } }
I20260925 10:37:08.456310 20285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:08.457407 20293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:08.462297 20293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:08.469985 20293 catalog_manager.cc:1347] Generated new cluster ID: da3321dab7d349439b26e10cba4e4612
I20260925 10:37:08.470131 20293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:08.483076 20293 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:08.484210 20293 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:08.491138 20293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc: Generated new TSK 0
I20260925 10:37:08.491894 20293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:08.497906 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44833
--builtin_ntp_servers=127.0.97.20:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:37:08.844251 20303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:08.844506 20303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:08.844587 20303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:08.852137 20303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:08.852260 20303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:08.852387 20303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260925 10:37:08.861248 20303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:08.862464 20303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:08.864084 20303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:08.876634 20308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:08.876701 20309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:08.876905 20311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:08.877931 20303 server_base.cc:1034] running on GCE node
I20260925 10:37:08.878621 20303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:08.879551 20303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:08.880805 20303 hybrid_clock.cc:648] HybridClock initialized: now 1790332628880764 us; error 29 us; skew 500 ppm
I20260925 10:37:08.881119 20303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:37:08.883450 20303 webserver.cc:460] Webserver started at http://127.0.97.1:34327/ using document root <none> and password file <none>
I20260925 10:37:08.884123 20303 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:08.884238 20303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:08.884548 20303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:08.887452 20303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/data/instance:
uuid: "a0a6954a64364fd5b2093ecbb6b764aa"
format_stamp: "Formatted at 2026-09-25 10:37:08 on dist-test-slave-dxdd"
server_key: "d2b599976f253a3112dabfb852150004"
server_key_iv: "88d63d414b8c0fb982c812815ba711c5"
server_key_version: "encryptionkey@0"
I20260925 10:37:08.888265 20303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a0a6954a64364fd5b2093ecbb6b764aa"
format_stamp: "Formatted at 2026-09-25 10:37:08 on dist-test-slave-dxdd"
server_key: "d2b599976f253a3112dabfb852150004"
server_key_iv: "88d63d414b8c0fb982c812815ba711c5"
server_key_version: "encryptionkey@0"
I20260925 10:37:08.893313 20303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 10:37:08.896586 20317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:08.898051 20303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:37:08.898232 20303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/wal
uuid: "a0a6954a64364fd5b2093ecbb6b764aa"
format_stamp: "Formatted at 2026-09-25 10:37:08 on dist-test-slave-dxdd"
server_key: "d2b599976f253a3112dabfb852150004"
server_key_iv: "88d63d414b8c0fb982c812815ba711c5"
server_key_version: "encryptionkey@0"
I20260925 10:37:08.898505 20303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:08.914705 20303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:08.915429 20303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:08.915690 20303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:08.916556 20303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:37:08.917960 20303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:37:08.918107 20303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:08.918243 20303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:37:08.918350 20303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:08.950826 20303 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45127
I20260925 10:37:08.950939 20429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45127 every 8 connection(s)
I20260925 10:37:08.952083 20303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/data/info.pb
I20260925 10:37:08.957737 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 20303
I20260925 10:37:08.957895 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-0/wal/instance
I20260925 10:37:08.958484 388 external_mini_cluster.cc:1428] Setting key f89fb3bd450f101b38f09592783f2a2e
I20260925 10:37:08.961439 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44833
--builtin_ntp_servers=127.0.97.20:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:37:08.965798 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:08.954666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58937 (local address 127.0.97.62:44833)
0925 10:37:08.955184 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:08.955189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:08.956175 (+ 986us) server_negotiation.cc:407] Connection header received
0925 10:37:08.958202 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:08.958206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:08.958271 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:08.958419 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:08.959695 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:08.960724 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:08.963185 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:08.963566 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:08.963659 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:08.965134 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:08.965139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:08.965142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:08.965158 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:08.965196 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:08.965207 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:08.965272 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:08.965507 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 10:37:08.965576 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":242,"threads_started":1}
I20260925 10:37:08.967192 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:08.955144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44833 (local address 127.0.97.1:58937)
0925 10:37:08.956014 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:08.956052 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:08.957917 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:08.958664 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:08.958676 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:08.958790 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:08.959542 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:08.959561 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:08.960872 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:08.960878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:08.963041 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:08.963060 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:08.963784 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:08.963794 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:08.964112 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:08.964822 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:37:08.964827 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:08.964876 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:37:08.964967 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:08.965298 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:08.965311 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:08.965320 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:37:08.965494 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 10:37:08.965888 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":293,"threads_started":1}
I20260925 10:37:08.968611 20430 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44833
I20260925 10:37:08.968896 20430 heartbeater.cc:464] Registering TS with master...
I20260925 10:37:08.969645 20430 heartbeater.cc:511] Master 127.0.97.62:44833 requested a full tablet report, sending...
I20260925 10:37:08.971810 20244 ts_manager.cc:194] Registered new tserver with Master: a0a6954a64364fd5b2093ecbb6b764aa (127.0.97.1:45127)
I20260925 10:37:08.973953 20244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:58937
W20260925 10:37:09.280795 20434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:09.281049 20434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:09.281132 20434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:09.288072 20434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:09.288163 20434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:09.288264 20434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260925 10:37:09.296931 20434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:09.298075 20434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:09.299774 20434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:09.312004 20439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:09.312033 20440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:09.311970 20442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:09.312915 20434 server_base.cc:1034] running on GCE node
I20260925 10:37:09.313514 20434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:09.314406 20434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:09.315608 20434 hybrid_clock.cc:648] HybridClock initialized: now 1790332629315576 us; error 26 us; skew 500 ppm
I20260925 10:37:09.315910 20434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:37:09.317970 20434 webserver.cc:460] Webserver started at http://127.0.97.2:36323/ using document root <none> and password file <none>
I20260925 10:37:09.318703 20434 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:09.318823 20434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:09.319114 20434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:09.321822 20434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/data/instance:
uuid: "f657c63a464a402e9340245225c0692f"
format_stamp: "Formatted at 2026-09-25 10:37:09 on dist-test-slave-dxdd"
server_key: "0ff742bbb3281a000970ac70f273257c"
server_key_iv: "24253e3c150b57f656dc5702a6203538"
server_key_version: "encryptionkey@0"
I20260925 10:37:09.322587 20434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f657c63a464a402e9340245225c0692f"
format_stamp: "Formatted at 2026-09-25 10:37:09 on dist-test-slave-dxdd"
server_key: "0ff742bbb3281a000970ac70f273257c"
server_key_iv: "24253e3c150b57f656dc5702a6203538"
server_key_version: "encryptionkey@0"
I20260925 10:37:09.327216 20434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 10:37:09.330582 20448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:09.331820 20434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:37:09.332003 20434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/wal
uuid: "f657c63a464a402e9340245225c0692f"
format_stamp: "Formatted at 2026-09-25 10:37:09 on dist-test-slave-dxdd"
server_key: "0ff742bbb3281a000970ac70f273257c"
server_key_iv: "24253e3c150b57f656dc5702a6203538"
server_key_version: "encryptionkey@0"
I20260925 10:37:09.332203 20434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:09.352293 20434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:09.353063 20434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:09.353340 20434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:09.354265 20434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:37:09.355728 20434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:37:09.355810 20434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:09.355880 20434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:37:09.355933 20434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:09.388257 20434 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45661
I20260925 10:37:09.388382 20560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45661 every 8 connection(s)
I20260925 10:37:09.389564 20434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/data/info.pb
I20260925 10:37:09.390110 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 20434
I20260925 10:37:09.390291 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/wal/instance
I20260925 10:37:09.390913 388 external_mini_cluster.cc:1428] Setting key 25dd68919902302a235a865ad8590f56
I20260925 10:37:09.393983 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44833
--builtin_ntp_servers=127.0.97.20:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:37:09.405120 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:09.392309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41243 (local address 127.0.97.62:44833)
0925 10:37:09.392474 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:09.392478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:09.394203 (+ 1725us) server_negotiation.cc:407] Connection header received
0925 10:37:09.397049 (+ 2846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:09.397057 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:09.397142 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:09.397209 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:09.398749 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.399452 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.401914 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.402458 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.402556 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:09.404241 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:09.404249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:09.404256 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:09.404281 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:09.404323 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:09.404341 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:09.404434 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:09.404642 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 10:37:09.404744 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:37:09.406009 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:09.392965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44833 (local address 127.0.97.2:41243)
0925 10:37:09.394041 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:09.394090 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:09.396751 (+ 2661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:09.397692 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:09.397705 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:09.397828 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:09.398590 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:09.398615 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.399605 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.399612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:09.401749 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:09.401766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.402668 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.402695 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:09.403034 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:09.403934 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:37:09.403937 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:09.403975 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:37:09.404051 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:09.404439 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:09.404446 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:09.404452 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:37:09.404605 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 10:37:09.404877 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":313,"threads_started":1}
I20260925 10:37:09.407444 20561 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44833
I20260925 10:37:09.407763 20561 heartbeater.cc:464] Registering TS with master...
I20260925 10:37:09.408450 20561 heartbeater.cc:511] Master 127.0.97.62:44833 requested a full tablet report, sending...
I20260925 10:37:09.409869 20244 ts_manager.cc:194] Registered new tserver with Master: f657c63a464a402e9340245225c0692f (127.0.97.2:45661)
I20260925 10:37:09.411087 20244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:41243
W20260925 10:37:09.709573 20564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:09.709815 20564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:09.709913 20564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:09.716807 20564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:09.716888 20564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:09.716984 20564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260925 10:37:09.725396 20564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:09.726531 20564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:09.728106 20564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:09.740320 20570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:09.740381 20569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:09.740402 20572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:09.741091 20564 server_base.cc:1034] running on GCE node
I20260925 10:37:09.741639 20564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:09.742578 20564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:09.743810 20564 hybrid_clock.cc:648] HybridClock initialized: now 1790332629743792 us; error 63 us; skew 500 ppm
I20260925 10:37:09.744109 20564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:37:09.746133 20564 webserver.cc:460] Webserver started at http://127.0.97.3:37129/ using document root <none> and password file <none>
I20260925 10:37:09.746830 20564 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:09.746953 20564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:09.747267 20564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:09.750057 20564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/data/instance:
uuid: "89a7d3f0498c4183ad3e34816bfba20f"
format_stamp: "Formatted at 2026-09-25 10:37:09 on dist-test-slave-dxdd"
server_key: "2f8ff27fa8a4627b292025b97a0733e2"
server_key_iv: "a45b8ad3ce294cb12c79a68dafdb0183"
server_key_version: "encryptionkey@0"
I20260925 10:37:09.750869 20564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/wal/instance:
uuid: "89a7d3f0498c4183ad3e34816bfba20f"
format_stamp: "Formatted at 2026-09-25 10:37:09 on dist-test-slave-dxdd"
server_key: "2f8ff27fa8a4627b292025b97a0733e2"
server_key_iv: "a45b8ad3ce294cb12c79a68dafdb0183"
server_key_version: "encryptionkey@0"
I20260925 10:37:09.755807 20564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:37:09.759173 20578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:09.760380 20564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:37:09.760553 20564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/wal
uuid: "89a7d3f0498c4183ad3e34816bfba20f"
format_stamp: "Formatted at 2026-09-25 10:37:09 on dist-test-slave-dxdd"
server_key: "2f8ff27fa8a4627b292025b97a0733e2"
server_key_iv: "a45b8ad3ce294cb12c79a68dafdb0183"
server_key_version: "encryptionkey@0"
I20260925 10:37:09.760787 20564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:09.784996 20564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:09.785760 20564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:09.786026 20564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:09.786944 20564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:37:09.788288 20564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:37:09.788407 20564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:09.788540 20564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:37:09.788631 20564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:09.819947 20564 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33175
I20260925 10:37:09.820027 20690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33175 every 8 connection(s)
I20260925 10:37:09.821201 20564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/data/info.pb
I20260925 10:37:09.831072 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 20564
I20260925 10:37:09.831265 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/wal/instance
I20260925 10:37:09.831703 388 external_mini_cluster.cc:1428] Setting key 05a5d855828e4851030a0f93502d19c8
I20260925 10:37:09.833369 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:09.823634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43323 (local address 127.0.97.62:44833)
0925 10:37:09.823803 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:09.823807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:09.825080 (+ 1273us) server_negotiation.cc:407] Connection header received
0925 10:37:09.826898 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:09.826904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:09.826992 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:09.827102 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:09.828105 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.828883 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.830421 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.831005 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.831149 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:09.832559 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:09.832564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:09.832586 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:09.832604 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:09.832638 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:09.832650 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:09.832753 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:09.832962 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 10:37:09.833056 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:37:09.834277 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:09.824077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44833 (local address 127.0.97.3:43323)
0925 10:37:09.824922 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:09.824961 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:09.826634 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:09.827328 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:09.827335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:09.827417 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:09.827929 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:09.827960 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.829038 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.829043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:09.830214 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:09.830225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.831250 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.831259 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:09.831532 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:09.832224 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:37:09.832227 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:09.832275 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:37:09.832354 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:09.832744 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:09.832754 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:09.832763 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:37:09.832949 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 10:37:09.833311 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":276,"threads_started":1}
I20260925 10:37:09.835520 20691 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44833
I20260925 10:37:09.835847 20691 heartbeater.cc:464] Registering TS with master...
I20260925 10:37:09.836565 20691 heartbeater.cc:511] Master 127.0.97.62:44833 requested a full tablet report, sending...
I20260925 10:37:09.837826 20244 ts_manager.cc:194] Registered new tserver with Master: 89a7d3f0498c4183ad3e34816bfba20f (127.0.97.3:33175)
I20260925 10:37:09.838647 20244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:43323
I20260925 10:37:09.846433 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:37:09.856282 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:09.850558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38946 (local address 127.0.97.62:44833)
0925 10:37:09.850786 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:09.850793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:09.851146 (+ 353us) server_negotiation.cc:407] Connection header received
0925 10:37:09.851332 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:09.851337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:09.851418 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:09.851523 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:09.851973 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.852653 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.853988 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.854464 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.854593 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:09.855588 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:09.855593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:09.855598 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:09.855621 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:09.855650 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:09.855662 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:09.855754 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:09.855891 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 10:37:09.855993 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:37:09.859700 20244 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:37:09.866482 20244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:37:09.904050 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:09.876970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45127 (local address 127.0.0.1:48892)
0925 10:37:09.894052 (+ 17082us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:09.894118 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:09.894478 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:09.895663 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:09.895669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:09.895709 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:09.896155 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:09.896181 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.898352 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.898359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:09.900810 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:09.900829 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.903167 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.903173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:09.903372 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:09.903420 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 10:37:09.903542 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 10:37:09.903735 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16736,"thread_start_us":16586,"threads_started":1}
I20260925 10:37:09.904978 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:09.877227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.0.97.1:45127)
0925 10:37:09.892524 (+ 15297us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:09.892537 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:09.894274 (+ 1737us) server_negotiation.cc:407] Connection header received
0925 10:37:09.894763 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:09.894775 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:09.895088 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:09.895286 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:09.896405 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.898198 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.901089 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.902617 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.902789 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:09.903046 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:09.904451 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0925 10:37:09.904629 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15054,"thread_start_us":15312,"threads_started":1}
I20260925 10:37:09.905265 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:09.877835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45661 (local address 127.0.0.1:43086)
0925 10:37:09.898010 (+ 20175us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:09.898034 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:09.898152 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:09.898714 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:09.898718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:09.898742 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:09.899141 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:09.899153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.900988 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.900996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:09.902760 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:09.902773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.904538 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.904544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:09.904736 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:09.904769 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 10:37:09.904863 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 10:37:09.904976 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20067,"thread_start_us":19931,"threads_started":1}
I20260925 10:37:09.905629 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:09.878507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.0.97.3:33175)
0925 10:37:09.893368 (+ 14861us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:09.893379 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:09.896984 (+ 3605us) server_negotiation.cc:407] Connection header received
0925 10:37:09.897275 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:09.897282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:09.897471 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:09.897624 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:09.898651 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.900092 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.902833 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.904200 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.904294 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:09.904555 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:09.905107 (+ 552us) server_negotiation.cc:299] Negotiation successful
0925 10:37:09.905276 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14635,"thread_start_us":14448,"threads_started":1}
I20260925 10:37:09.905691 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:09.878024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.0.97.2:45661)
0925 10:37:09.896292 (+ 18268us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:09.896304 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:09.898165 (+ 1861us) server_negotiation.cc:407] Connection header received
0925 10:37:09.898297 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:09.898327 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:09.898509 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:09.898724 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:09.899353 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.900846 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.902980 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.904340 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.904485 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:09.904759 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:09.904900 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 10:37:09.905048 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18055,"thread_start_us":18279,"threads_started":1}
I20260925 10:37:09.906505 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:09.877562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33175 (local address 127.0.0.1:37946)
0925 10:37:09.896875 (+ 19313us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:09.896895 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:09.897076 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:09.897958 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:09.897963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:09.897985 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:09.898459 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:09.898473 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.900286 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.900295 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:09.902600 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:09.902613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:09.904725 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:09.904732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:09.904895 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:09.904926 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 10:37:09.906032 (+ 1106us) client_negotiation.cc:240] Negotiation successful
0925 10:37:09.906151 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19206,"thread_start_us":19072,"threads_started":1}
I20260925 10:37:09.908846 20365 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce10a322d884d5ea6861d0b1fa25d2b (DEFAULT_TABLE table=test-table [id=096b8abde9f2424395533aeb98102e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:37:09.908850 20496 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce10a322d884d5ea6861d0b1fa25d2b (DEFAULT_TABLE table=test-table [id=096b8abde9f2424395533aeb98102e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:37:09.910084 20496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce10a322d884d5ea6861d0b1fa25d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:09.910091 20365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce10a322d884d5ea6861d0b1fa25d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:09.912384 20626 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce10a322d884d5ea6861d0b1fa25d2b (DEFAULT_TABLE table=test-table [id=096b8abde9f2424395533aeb98102e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:37:09.913569 20626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce10a322d884d5ea6861d0b1fa25d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:09.921676 20709 tablet_bootstrap.cc:492] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f: Bootstrap starting.
I20260925 10:37:09.924594 20709 tablet_bootstrap.cc:654] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:09.925561 20709 log.cc:824] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:09.928426 20712 tablet_bootstrap.cc:492] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f: Bootstrap starting.
I20260925 10:37:09.928903 20709 tablet_bootstrap.cc:492] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f: No bootstrap required, opened a new log
I20260925 10:37:09.929170 20709 ts_tablet_manager.cc:1397] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260925 10:37:09.931842 20712 tablet_bootstrap.cc:654] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:09.933012 20712 log.cc:824] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:09.934130 20711 tablet_bootstrap.cc:492] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa: Bootstrap starting.
I20260925 10:37:09.937085 20712 tablet_bootstrap.cc:492] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f: No bootstrap required, opened a new log
I20260925 10:37:09.937343 20712 ts_tablet_manager.cc:1397] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 10:37:09.937568 20711 tablet_bootstrap.cc:654] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:09.938781 20711 log.cc:824] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:09.939713 20709 raft_consensus.cc:348] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a6954a64364fd5b2093ecbb6b764aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45127 } } peers { permanent_uuid: "89a7d3f0498c4183ad3e34816bfba20f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33175 } } peers { permanent_uuid: "f657c63a464a402e9340245225c0692f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45661 } }
I20260925 10:37:09.940092 20709 raft_consensus.cc:374] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:09.940209 20709 raft_consensus.cc:729] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f657c63a464a402e9340245225c0692f, State: Initialized, Role: FOLLOWER
I20260925 10:37:09.940838 20709 consensus_queue.cc:260] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a6954a64364fd5b2093ecbb6b764aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45127 } } peers { permanent_uuid: "89a7d3f0498c4183ad3e34816bfba20f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33175 } } peers { permanent_uuid: "f657c63a464a402e9340245225c0692f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45661 } }
I20260925 10:37:09.941890 20711 tablet_bootstrap.cc:492] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa: No bootstrap required, opened a new log
I20260925 10:37:09.942211 20711 ts_tablet_manager.cc:1397] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 10:37:09.944948 20712 raft_consensus.cc:348] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a6954a64364fd5b2093ecbb6b764aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45127 } } peers { permanent_uuid: "89a7d3f0498c4183ad3e34816bfba20f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33175 } } peers { permanent_uuid: "f657c63a464a402e9340245225c0692f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45661 } }
I20260925 10:37:09.945293 20712 raft_consensus.cc:374] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:09.945449 20712 raft_consensus.cc:729] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89a7d3f0498c4183ad3e34816bfba20f, State: Initialized, Role: FOLLOWER
I20260925 10:37:09.945955 20561 heartbeater.cc:503] Master 127.0.97.62:44833 was elected leader, sending a full tablet report...
I20260925 10:37:09.946580 20709 ts_tablet_manager.cc:1428] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f: Time spent starting tablet: real 0.017s user 0.013s sys 0.006s
I20260925 10:37:09.946941 20712 consensus_queue.cc:260] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a6954a64364fd5b2093ecbb6b764aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45127 } } peers { permanent_uuid: "89a7d3f0498c4183ad3e34816bfba20f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33175 } } peers { permanent_uuid: "f657c63a464a402e9340245225c0692f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45661 } }
I20260925 10:37:09.950289 20691 heartbeater.cc:503] Master 127.0.97.62:44833 was elected leader, sending a full tablet report...
I20260925 10:37:09.950995 20712 ts_tablet_manager.cc:1428] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260925 10:37:09.956017 20711 raft_consensus.cc:348] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a6954a64364fd5b2093ecbb6b764aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45127 } } peers { permanent_uuid: "89a7d3f0498c4183ad3e34816bfba20f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33175 } } peers { permanent_uuid: "f657c63a464a402e9340245225c0692f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45661 } }
I20260925 10:37:09.956455 20711 raft_consensus.cc:374] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:09.956557 20711 raft_consensus.cc:729] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a6954a64364fd5b2093ecbb6b764aa, State: Initialized, Role: FOLLOWER
I20260925 10:37:09.957213 20711 consensus_queue.cc:260] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a6954a64364fd5b2093ecbb6b764aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45127 } } peers { permanent_uuid: "89a7d3f0498c4183ad3e34816bfba20f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33175 } } peers { permanent_uuid: "f657c63a464a402e9340245225c0692f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45661 } }
I20260925 10:37:09.958346 20430 heartbeater.cc:503] Master 127.0.97.62:44833 was elected leader, sending a full tablet report...
I20260925 10:37:09.958853 20711 ts_tablet_manager.cc:1428] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
W20260925 10:37:10.072968 20692 tablet.cc:2369] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:37:10.141744 20562 tablet.cc:2369] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:37:10.204442 20431 tablet.cc:2369] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:37:10.264863 20715 raft_consensus.cc:482] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:37:10.265223 20715 raft_consensus.cc:504] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a6954a64364fd5b2093ecbb6b764aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45127 } } peers { permanent_uuid: "89a7d3f0498c4183ad3e34816bfba20f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33175 } } peers { permanent_uuid: "f657c63a464a402e9340245225c0692f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45661 } }
I20260925 10:37:10.266846 20715 leader_election.cc:288] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0a6954a64364fd5b2093ecbb6b764aa (127.0.97.1:45127), 89a7d3f0498c4183ad3e34816bfba20f (127.0.97.3:33175)
I20260925 10:37:10.275944 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:10.267297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40687 (local address 127.0.97.3:33175)
0925 10:37:10.267451 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:10.267457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:10.268145 (+ 688us) server_negotiation.cc:407] Connection header received
0925 10:37:10.268372 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:10.268378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:10.268461 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:10.268558 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:10.269426 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:10.270531 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:10.273568 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:10.274789 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:10.274910 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:10.275010 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:10.275565 (+ 555us) server_negotiation.cc:299] Negotiation successful
0925 10:37:10.275691 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:37:10.275943 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:10.267251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58485 (local address 127.0.97.1:45127)
0925 10:37:10.267442 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:10.267448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:10.268152 (+ 704us) server_negotiation.cc:407] Connection header received
0925 10:37:10.268372 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:10.268378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:10.268467 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:10.268558 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:37:10.269426 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:10.270488 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:10.273569 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:10.274793 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:10.274910 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:10.275004 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:10.275565 (+ 561us) server_negotiation.cc:299] Negotiation successful
0925 10:37:10.275698 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:37:10.276170 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:10.267182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45127 (local address 127.0.97.2:58485)
0925 10:37:10.268041 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:10.268058 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:10.268177 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:10.268598 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:10.268602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:10.268648 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:10.269234 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:10.269262 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:10.270668 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:10.270673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:10.273352 (+ 2679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:10.273367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:10.275016 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:10.275023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:10.275318 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:10.275430 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 10:37:10.275525 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 10:37:10.275739 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":364,"spinlock_wait_cycles":14336,"thread_start_us":493,"threads_started":1}
I20260925 10:37:10.276193 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:10.267100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33175 (local address 127.0.97.2:40687)
0925 10:37:10.268004 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:10.268028 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:10.268211 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:10.268598 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:10.268602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:10.268649 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:37:10.269295 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:10.269304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:10.270668 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:10.270673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:10.273360 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:10.273369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:10.275024 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:10.275029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:10.275313 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:10.275432 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 10:37:10.275525 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 10:37:10.275739 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":387,"spinlock_wait_cycles":45440,"thread_start_us":529,"threads_started":1}
I20260925 10:37:10.277299 20385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce10a322d884d5ea6861d0b1fa25d2b" candidate_uuid: "f657c63a464a402e9340245225c0692f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0a6954a64364fd5b2093ecbb6b764aa" is_pre_election: true
I20260925 10:37:10.277336 20646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce10a322d884d5ea6861d0b1fa25d2b" candidate_uuid: "f657c63a464a402e9340245225c0692f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89a7d3f0498c4183ad3e34816bfba20f" is_pre_election: true
I20260925 10:37:10.277787 20385 raft_consensus.cc:2454] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f657c63a464a402e9340245225c0692f in term 0.
I20260925 10:37:10.277797 20646 raft_consensus.cc:2454] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f657c63a464a402e9340245225c0692f in term 0.
I20260925 10:37:10.278506 20451 leader_election.cc:302] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 89a7d3f0498c4183ad3e34816bfba20f, a0a6954a64364fd5b2093ecbb6b764aa, f657c63a464a402e9340245225c0692f; no voters:
I20260925 10:37:10.278925 20715 raft_consensus.cc:2781] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:37:10.279064 20715 raft_consensus.cc:482] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:37:10.279181 20715 raft_consensus.cc:3037] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:10.281302 20715 raft_consensus.cc:504] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a6954a64364fd5b2093ecbb6b764aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45127 } } peers { permanent_uuid: "89a7d3f0498c4183ad3e34816bfba20f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33175 } } peers { permanent_uuid: "f657c63a464a402e9340245225c0692f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45661 } }
I20260925 10:37:10.281868 20715 leader_election.cc:288] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [CANDIDATE]: Term 1 election: Requested vote from peers a0a6954a64364fd5b2093ecbb6b764aa (127.0.97.1:45127), 89a7d3f0498c4183ad3e34816bfba20f (127.0.97.3:33175)
I20260925 10:37:10.282370 20646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce10a322d884d5ea6861d0b1fa25d2b" candidate_uuid: "f657c63a464a402e9340245225c0692f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89a7d3f0498c4183ad3e34816bfba20f"
I20260925 10:37:10.282374 20385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce10a322d884d5ea6861d0b1fa25d2b" candidate_uuid: "f657c63a464a402e9340245225c0692f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0a6954a64364fd5b2093ecbb6b764aa"
I20260925 10:37:10.282573 20385 raft_consensus.cc:3037] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:10.282567 20646 raft_consensus.cc:3037] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:10.284744 20385 raft_consensus.cc:2454] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f657c63a464a402e9340245225c0692f in term 1.
I20260925 10:37:10.284808 20646 raft_consensus.cc:2454] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f657c63a464a402e9340245225c0692f in term 1.
I20260925 10:37:10.285127 20449 leader_election.cc:302] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0a6954a64364fd5b2093ecbb6b764aa, f657c63a464a402e9340245225c0692f; no voters:
I20260925 10:37:10.285451 20715 raft_consensus.cc:2781] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:10.285960 20715 raft_consensus.cc:686] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [term 1 LEADER]: Becoming Leader. State: Replica: f657c63a464a402e9340245225c0692f, State: Running, Role: LEADER
I20260925 10:37:10.286469 20715 consensus_queue.cc:237] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a6954a64364fd5b2093ecbb6b764aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45127 } } peers { permanent_uuid: "89a7d3f0498c4183ad3e34816bfba20f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33175 } } peers { permanent_uuid: "f657c63a464a402e9340245225c0692f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45661 } }
I20260925 10:37:10.290994 20243 catalog_manager.cc:5495] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f reported cstate change: term changed from 0 to 1, leader changed from <none> to f657c63a464a402e9340245225c0692f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f657c63a464a402e9340245225c0692f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a6954a64364fd5b2093ecbb6b764aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89a7d3f0498c4183ad3e34816bfba20f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f657c63a464a402e9340245225c0692f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45661 } health_report { overall_health: HEALTHY } } }
I20260925 10:37:10.360493 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:10.355259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.0.97.2:45661)
0925 10:37:10.355487 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:10.355491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:10.355508 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:37:10.355673 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:10.355676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:10.355750 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:10.355864 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:10.356144 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:10.357001 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:10.358470 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:10.358976 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:10.359066 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:10.359489 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:10.359967 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:10.360079 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:10.360139 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 10:37:10.360204 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:37:10.368971 20385 raft_consensus.cc:1261] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa [term 1 FOLLOWER]: Refusing update from remote peer f657c63a464a402e9340245225c0692f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:37:10.368970 20646 raft_consensus.cc:1261] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f [term 1 FOLLOWER]: Refusing update from remote peer f657c63a464a402e9340245225c0692f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:37:10.370085 20720 consensus_queue.cc:1035] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a7d3f0498c4183ad3e34816bfba20f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:37:10.370536 20715 consensus_queue.cc:1035] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0a6954a64364fd5b2093ecbb6b764aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:37:10.384094 20725 mvcc.cc:205] Tried to move back new op lower bound from 7333202453981532160 to 7333202453656879104. Current Snapshot: MvccSnapshot[applied={T|T < 7333202453981532160}]
I20260925 10:37:10.402127 20243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:37:10.402464 20243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:37:10.406257 20243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f2312e3c46744fd84f08dbe02895afc: Sending DeleteTablet for 3 replicas of tablet 3ce10a322d884d5ea6861d0b1fa25d2b
I20260925 10:37:10.407296 20365 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ce10a322d884d5ea6861d0b1fa25d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:37:10 UTC) from {username='slave'} at 127.0.0.1:48892
I20260925 10:37:10.407670 20626 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ce10a322d884d5ea6861d0b1fa25d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:37:10 UTC) from {username='slave'} at 127.0.0.1:37946
I20260925 10:37:10.408129 20496 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ce10a322d884d5ea6861d0b1fa25d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:37:10 UTC) from {username='slave'} at 127.0.0.1:43086
I20260925 10:37:10.408828 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 20303
I20260925 10:37:10.435843 20735 tablet_replica.cc:330] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f: stopping tablet replica
I20260925 10:37:10.436411 20735 raft_consensus.cc:2229] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:37:10.436818 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:10.410275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48908 (local address 127.0.97.1:45127)
0925 10:37:10.411826 (+ 1551us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:10.411835 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:10.426171 (+ 14336us) server_negotiation.cc:407] Connection header received
0925 10:37:10.426668 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:10.426677 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:10.426847 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:10.426939 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:10.427739 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:10.429214 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:10.431823 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:10.432738 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:10.432856 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:10.434622 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:10.434637 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:10.434658 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:10.435565 (+ 907us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:10.435668 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:10.435710 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:10.435813 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:10.436139 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 10:37:10.436281 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1401}
I20260925 10:37:10.437517 20733 tablet_replica.cc:330] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa: stopping tablet replica
I20260925 10:37:10.438030 20425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:10.438066 20735 raft_consensus.cc:2258] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:37:10.440137 20735 ts_tablet_manager.cc:1905] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:37:10.450759 20734 tablet_replica.cc:330] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f: stopping tablet replica
I20260925 10:37:10.451145 20734 raft_consensus.cc:2229] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:37:10.451797 20734 raft_consensus.cc:2258] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:37:10.451866 20735 ts_tablet_manager.cc:1918] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:37:10.452060 20735 log.cc:1196] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-1/wal/wals/3ce10a322d884d5ea6861d0b1fa25d2b
I20260925 10:37:10.452710 20735 ts_tablet_manager.cc:1939] T 3ce10a322d884d5ea6861d0b1fa25d2b P f657c63a464a402e9340245225c0692f: Deleting consensus metadata
I20260925 10:37:10.453969 20734 ts_tablet_manager.cc:1905] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:37:10.454581 20229 catalog_manager.cc:4841] TS f657c63a464a402e9340245225c0692f (127.0.97.2:45661): tablet 3ce10a322d884d5ea6861d0b1fa25d2b (table test-table [id=096b8abde9f2424395533aeb98102e93]) successfully deleted
I20260925 10:37:10.459092 20734 ts_tablet_manager.cc:1918] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:37:10.459267 20734 log.cc:1196] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f: Deleting WAL directory at /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332315635355-388-0/minicluster-data/ts-2/wal/wals/3ce10a322d884d5ea6861d0b1fa25d2b
I20260925 10:37:10.459692 20734 ts_tablet_manager.cc:1939] T 3ce10a322d884d5ea6861d0b1fa25d2b P 89a7d3f0498c4183ad3e34816bfba20f: Deleting consensus metadata
I20260925 10:37:10.461038 20230 catalog_manager.cc:4841] TS 89a7d3f0498c4183ad3e34816bfba20f (127.0.97.3:33175): tablet 3ce10a322d884d5ea6861d0b1fa25d2b (table test-table [id=096b8abde9f2424395533aeb98102e93]) successfully deleted
I20260925 10:37:10.438066 20733 raft_consensus.cc:2229] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:37:10.552709 20733 raft_consensus.cc:2258] T 3ce10a322d884d5ea6861d0b1fa25d2b P a0a6954a64364fd5b2093ecbb6b764aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:37:10.553486 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 20303
W20260925 10:37:10.571146 20228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45127 (error 108)
I20260925 10:37:10.572362 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 20434
I20260925 10:37:10.573244 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:10.572330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45127 (local address 127.0.0.1:48918)
0925 10:37:10.572868 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:10.573016 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":272,"threads_started":1}
W20260925 10:37:10.573637 20228 catalog_manager.cc:4584] TS a0a6954a64364fd5b2093ecbb6b764aa (127.0.97.1:45127): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ce10a322d884d5ea6861d0b1fa25d2b: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45127: connect: Connection refused (error 111)
I20260925 10:37:10.586742 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:10.573252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43092 (local address 127.0.97.2:45661)
0925 10:37:10.573424 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:10.573429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:10.573451 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:37:10.573538 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:10.573542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:10.573642 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:10.573778 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:10.574243 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:10.575462 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:10.576689 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:10.577414 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:10.577543 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:10.585072 (+ 7529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:10.585084 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:10.585102 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:10.585915 (+ 813us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:10.585975 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:10.586000 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:10.586103 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:10.586238 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 10:37:10.586365 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:37:10.587499 20556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:10.627317 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:10.626855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45127 (local address 127.0.0.1:48932)
0925 10:37:10.627080 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:10.627179 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 10:37:10.628597 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:10.628168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45127 (local address 127.0.0.1:48938)
0925 10:37:10.628352 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:10.628427 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 10:37:10.689443 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:10.688896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45127 (local address 127.0.0.1:48948)
0925 10:37:10.689189 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:10.689291 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260925 10:37:10.690519 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:10.690135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45127 (local address 127.0.0.1:48952)
0925 10:37:10.690272 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:10.690393 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 10:37:10.703855 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 20434
I20260925 10:37:10.723127 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 20564
I20260925 10:37:10.729197 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:10.723780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.0.97.3:33175)
0925 10:37:10.723953 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:10.723957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:10.723989 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:37:10.724116 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:10.724120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:10.724235 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:10.724320 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:10.724657 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:10.725350 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:10.726647 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:10.727011 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:10.727099 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:10.728048 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:10.728055 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:10.728069 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:10.728585 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:10.728624 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:10.728642 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:10.728728 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:10.728878 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 10:37:10.728951 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:37:10.729805 20686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:10.770963 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:10.770512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45127 (local address 127.0.0.1:48962)
0925 10:37:10.770738 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:10.770825 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 10:37:10.771976 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:10.771613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45127 (local address 127.0.0.1:48968)
0925 10:37:10.771789 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:10.771851 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260925 10:37:10.815802 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 20564
I20260925 10:37:10.831776 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 20213
I20260925 10:37:10.832785 20274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:10.939734 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 20213
2026-09-25T10:37:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3009 ms)
[----------] 6 tests from EncryptionPolicyTest (20961 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:11 dist-test-slave-dxdd krb5kdc[20745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:11 dist-test-slave-dxdd krb5kdc[20745](info): set up 2 sockets
Sep 25 10:37:11 dist-test-slave-dxdd krb5kdc[20745](info): commencing operation
krb5kdc: starting...
W20260925 10:37:12.058681 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:12 dist-test-slave-dxdd krb5kdc[20745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:37:12.294838 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:12.610837 20758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:12.611186 20758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:12.611384 20758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:12.618484 20758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:12.618637 20758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:12.618741 20758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:12.618848 20758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:12.618934 20758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:12.628552 20758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:12.629812 20758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:12.631114 20758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:12.642925 20764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:12.643025 20763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:12.643675 20766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:12.644389 20758 server_base.cc:1034] running on GCE node
I20260925 10:37:12.645052 20758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:12.646447 20758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:12.647645 20758 hybrid_clock.cc:648] HybridClock initialized: now 1790332632647594 us; error 41 us; skew 500 ppm
I20260925 10:37:12.647919 20758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:12 dist-test-slave-dxdd krb5kdc[20745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:12.650846 20758 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:37:12.652186 20758 webserver.cc:460] Webserver started at http://127.0.0.1:34309/ using document root <none> and password file <none>
I20260925 10:37:12.652839 20758 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:12.652978 20758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:12.653290 20758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:12.656279 20758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "d6a67230888144d0b234920edbdc2f3b"
format_stamp: "Formatted at 2026-09-25 10:37:12 on dist-test-slave-dxdd"
server_key: "bfc5b0c3e836c270e3a9ae536f9c5bbe"
server_key_iv: "fce5454971faebc8015abb4ee6c93bb5"
server_key_version: "encryptionkey@0"
I20260925 10:37:12.657121 20758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "d6a67230888144d0b234920edbdc2f3b"
format_stamp: "Formatted at 2026-09-25 10:37:12 on dist-test-slave-dxdd"
server_key: "bfc5b0c3e836c270e3a9ae536f9c5bbe"
server_key_iv: "fce5454971faebc8015abb4ee6c93bb5"
server_key_version: "encryptionkey@0"
I20260925 10:37:12.664060 20758 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260925 10:37:12.667240 20773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:12.668406 20758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 10:37:12.668532 20758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "d6a67230888144d0b234920edbdc2f3b"
format_stamp: "Formatted at 2026-09-25 10:37:12 on dist-test-slave-dxdd"
server_key: "bfc5b0c3e836c270e3a9ae536f9c5bbe"
server_key_iv: "fce5454971faebc8015abb4ee6c93bb5"
server_key_version: "encryptionkey@0"
I20260925 10:37:12.668699 20758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:12.696344 20758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:12.699926 20758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:12.700163 20758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:12.718778 20758 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42483
I20260925 10:37:12.718823 20824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42483 every 8 connection(s)
I20260925 10:37:12.720077 20758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:12.723603 20825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:12.727483 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 20758
I20260925 10:37:12.727681 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:12.728403 388 external_mini_cluster.cc:1428] Setting key 95ef9ae9c21ce85ac983847945b67194
I20260925 10:37:12.734282 20825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b: Bootstrap starting.
I20260925 10:37:12.737205 20825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:12.738143 20825 log.cc:824] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:12.741403 20825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b: No bootstrap required, opened a new log
Sep 25 10:37:12 dist-test-slave-dxdd krb5kdc[20745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:37:12.751969 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:12.730225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40751 (local address 127.0.0.1:42483)
0925 10:37:12.730928 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:12.730938 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:12.730974 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:37:12.733127 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:12.733147 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:12.733495 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:12.733845 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:12.735300 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:12.736641 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:12.737942 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:12.738893 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:12.739094 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:12.743710 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:12.743720 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:12.743737 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:12.743786 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:12.747680 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:12.748378 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:12.748387 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:12.748398 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:12.748616 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:12.749015 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:12.749019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:12.749023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:12.749667 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:12.749993 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:12.750360 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 10:37:12.750683 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":308,"threads_started":1}
I20260925 10:37:12.751936 20825 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a67230888144d0b234920edbdc2f3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42483 } }
I20260925 10:37:12.752375 20825 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:12.752478 20825 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a67230888144d0b234920edbdc2f3b, State: Initialized, Role: FOLLOWER
I20260925 10:37:12.753094 20825 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a67230888144d0b234920edbdc2f3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42483 } }
I20260925 10:37:12.753389 20825 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:12.753545 20825 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:12.753741 20825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:12.755839 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:12.756338 20825 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a67230888144d0b234920edbdc2f3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42483 } }
I20260925 10:37:12.756798 20825 leader_election.cc:302] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6a67230888144d0b234920edbdc2f3b; no voters:
I20260925 10:37:12.757467 20825 leader_election.cc:288] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:12.757579 20830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:12.758759 20830 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [term 1 LEADER]: Becoming Leader. State: Replica: d6a67230888144d0b234920edbdc2f3b, State: Running, Role: LEADER
I20260925 10:37:12.759348 20830 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a67230888144d0b234920edbdc2f3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42483 } }
I20260925 10:37:12.759801 20825 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:12.762508 20832 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6a67230888144d0b234920edbdc2f3b. Latest consensus state: current_term: 1 leader_uuid: "d6a67230888144d0b234920edbdc2f3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a67230888144d0b234920edbdc2f3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42483 } } }
I20260925 10:37:12.763564 20832 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:12.763168 20831 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6a67230888144d0b234920edbdc2f3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a67230888144d0b234920edbdc2f3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42483 } } }
I20260925 10:37:12.764227 20831 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:12.777669 20839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:12.784503 20839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:12.794926 20851 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:37:12.797334 20839 catalog_manager.cc:1347] Generated new cluster ID: b0e19dc221a2476e80e06f936b3060e1
I20260925 10:37:12.797430 20839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:12.805549 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:12.788193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60601 (local address 127.0.0.1:42483)
0925 10:37:12.788384 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:12.788390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:12.790052 (+ 1662us) server_negotiation.cc:407] Connection header received
0925 10:37:12.790265 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:12.790271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:12.790384 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:12.790495 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:12.792573 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:12.793792 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:12.796194 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:12.796840 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:12.796935 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:12.799630 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:12.799643 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:12.799647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:12.799675 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:12.802599 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:12.803386 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:12.803393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:12.803401 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:12.803541 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:12.804008 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:12.804012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:12.804014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:12.804334 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:12.804607 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:12.804996 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 10:37:12.805178 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:37:12.827456 20839 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:12.829056 20839 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:12.836421 20839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6a67230888144d0b234920edbdc2f3b: Generated new TSK 0
I20260925 10:37:12.837464 20839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:12.846217 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 20758
I20260925 10:37:12.847311 20820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:12.938416 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 20758
W20260925 10:37:12.959553 20743 connection.cc:535] client connection to 127.0.0.1:42483 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42483): Connection reset by peer (error 104)
2026-09-25T10:37:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1997 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:13 dist-test-slave-dxdd krb5kdc[20857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:13 dist-test-slave-dxdd krb5kdc[20857](info): set up 2 sockets
Sep 25 10:37:13 dist-test-slave-dxdd krb5kdc[20857](info): commencing operation
krb5kdc: starting...
W20260925 10:37:14.046666 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:14 dist-test-slave-dxdd krb5kdc[20857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:37:14.276688 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46713
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:14.593536 20870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:14.593798 20870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:14.593887 20870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:14.601029 20870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:14.601135 20870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:14.601182 20870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:14.601243 20870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:14.601286 20870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:14.610760 20870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46713
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:14.611909 20870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:14.613157 20870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:14.624727 20875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:14.624811 20876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:14.624787 20878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:14.625484 20870 server_base.cc:1034] running on GCE node
I20260925 10:37:14.626200 20870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:14.627570 20870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:14.628798 20870 hybrid_clock.cc:648] HybridClock initialized: now 1790332634628759 us; error 50 us; skew 500 ppm
I20260925 10:37:14.629168 20870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:14 dist-test-slave-dxdd krb5kdc[20857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:14.632138 20870 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:37:14.633510 20870 webserver.cc:460] Webserver started at http://127.0.0.1:44273/ using document root <none> and password file <none>
I20260925 10:37:14.634159 20870 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:14.634339 20870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:14.634661 20870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:14.637482 20870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "e172fc4c37bc4b29b41dd60334bc1311"
format_stamp: "Formatted at 2026-09-25 10:37:14 on dist-test-slave-dxdd"
server_key: "1d5d73137ab0e2eea8b6d3f980cf403f"
server_key_iv: "63ae5b996f08698fb42dc0ab74a8e52e"
server_key_version: "encryptionkey@0"
I20260925 10:37:14.638223 20870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "e172fc4c37bc4b29b41dd60334bc1311"
format_stamp: "Formatted at 2026-09-25 10:37:14 on dist-test-slave-dxdd"
server_key: "1d5d73137ab0e2eea8b6d3f980cf403f"
server_key_iv: "63ae5b996f08698fb42dc0ab74a8e52e"
server_key_version: "encryptionkey@0"
I20260925 10:37:14.643224 20870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 10:37:14.646620 20885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:14.647796 20870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:37:14.647989 20870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "e172fc4c37bc4b29b41dd60334bc1311"
format_stamp: "Formatted at 2026-09-25 10:37:14 on dist-test-slave-dxdd"
server_key: "1d5d73137ab0e2eea8b6d3f980cf403f"
server_key_iv: "63ae5b996f08698fb42dc0ab74a8e52e"
server_key_version: "encryptionkey@0"
I20260925 10:37:14.648185 20870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:14.662248 20870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:14.666391 20870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:14.666671 20870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:14.684514 20870 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46713
I20260925 10:37:14.684585 20936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46713 every 8 connection(s)
I20260925 10:37:14.686252 20870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:14.687237 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 20870
I20260925 10:37:14.687433 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:14.687969 388 external_mini_cluster.cc:1428] Setting key 37775939509ac8c4829cf9d3aae56a15
I20260925 10:37:14.690834 20937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:14.700733 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:14.689783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52125 (local address 127.0.0.1:46713)
0925 10:37:14.690598 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:14.690612 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:14.690652 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:37:14.692296 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:14.692317 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:14.692673 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:14.693055 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:14.693591 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:14.695020 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:14.696323 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:14.696964 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:14.697167 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:14.698545 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:14.698552 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:14.698568 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:14.699115 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:14.699191 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:14.699215 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:14.699386 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:14.699649 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 10:37:14.699799 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":355,"threads_started":1}
I20260925 10:37:14.701164 20937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311: Bootstrap starting.
I20260925 10:37:14.704430 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:14.704614 20937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:14.705528 20937 log.cc:824] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:14.708531 20937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311: No bootstrap required, opened a new log
I20260925 10:37:14.718168 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:14.710440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38829 (local address 127.0.0.1:46713)
0925 10:37:14.711572 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:14.711577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:14.711597 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:37:14.711693 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:14.711698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:14.711795 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:14.711903 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:14.712652 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:14.713470 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:14.714995 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:14.715814 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:14.715954 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:14.717343 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:14.717355 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:14.717359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:14.717375 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:14.717427 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:14.717442 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:14.717582 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:14.717770 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 10:37:14.717820 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 10:37:14.719511 20937 raft_consensus.cc:348] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e172fc4c37bc4b29b41dd60334bc1311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46713 } }
I20260925 10:37:14.719930 20937 raft_consensus.cc:374] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:14.720002 20937 raft_consensus.cc:729] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e172fc4c37bc4b29b41dd60334bc1311, State: Initialized, Role: FOLLOWER
I20260925 10:37:14.720557 20937 consensus_queue.cc:260] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e172fc4c37bc4b29b41dd60334bc1311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46713 } }
I20260925 10:37:14.720750 20937 raft_consensus.cc:388] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:14.720857 20937 raft_consensus.cc:482] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:14.720983 20937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:14.723343 20937 raft_consensus.cc:504] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e172fc4c37bc4b29b41dd60334bc1311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46713 } }
I20260925 10:37:14.723788 20937 leader_election.cc:302] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e172fc4c37bc4b29b41dd60334bc1311; no voters:
I20260925 10:37:14.724289 20937 leader_election.cc:288] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:14.724450 20947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:14.725466 20947 raft_consensus.cc:686] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [term 1 LEADER]: Becoming Leader. State: Replica: e172fc4c37bc4b29b41dd60334bc1311, State: Running, Role: LEADER
I20260925 10:37:14.725975 20947 consensus_queue.cc:237] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e172fc4c37bc4b29b41dd60334bc1311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46713 } }
I20260925 10:37:14.726455 20937 sys_catalog.cc:564] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:14.728549 20948 sys_catalog.cc:455] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e172fc4c37bc4b29b41dd60334bc1311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e172fc4c37bc4b29b41dd60334bc1311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46713 } } }
I20260925 10:37:14.728897 20948 sys_catalog.cc:458] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:14.729377 20949 sys_catalog.cc:455] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e172fc4c37bc4b29b41dd60334bc1311. Latest consensus state: current_term: 1 leader_uuid: "e172fc4c37bc4b29b41dd60334bc1311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e172fc4c37bc4b29b41dd60334bc1311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46713 } } }
I20260925 10:37:14.729684 20949 sys_catalog.cc:458] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:14.729787 20954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:14.737835 20954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:14.746910 20954 catalog_manager.cc:1347] Generated new cluster ID: d0ce70b10b1647dab0bcc2dcf893ea77
I20260925 10:37:14.747017 20954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:14.768664 20954 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:14.770521 20954 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:14.778604 20954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e172fc4c37bc4b29b41dd60334bc1311: Generated new TSK 0
I20260925 10:37:14.779623 20954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:14.789979 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 20870
W20260925 10:37:14.791848 20932 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52125
W20260925 10:37:14.792541 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 20870: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:37:14.792748 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 20870
W20260925 10:37:14.809073 20855 connection.cc:535] client connection to 127.0.0.1:46713 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46713): Connection reset by peer (error 104)
2026-09-25T10:37:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1849 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:14 dist-test-slave-dxdd krb5kdc[20968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:14 dist-test-slave-dxdd krb5kdc[20968](info): set up 2 sockets
Sep 25 10:37:14 dist-test-slave-dxdd krb5kdc[20968](info): commencing operation
krb5kdc: starting...
W20260925 10:37:15.882743 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:16 dist-test-slave-dxdd krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:37:16.125452 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:16.440815 20981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:16.441067 20981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:16.441159 20981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:16.447960 20981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:16.448046 20981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:16.448088 20981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:16.448134 20981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:16.448167 20981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:16.457274 20981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:16.458395 20981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:16.459537 20981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:16.470767 20989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:16.470811 20987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:16.470835 20986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:16.471925 20981 server_base.cc:1034] running on GCE node
I20260925 10:37:16.472532 20981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:16.473776 20981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:16.475005 20981 hybrid_clock.cc:648] HybridClock initialized: now 1790332636474964 us; error 52 us; skew 500 ppm
I20260925 10:37:16.475337 20981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:16 dist-test-slave-dxdd krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:16.482649 20981 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:37:16.484066 20981 webserver.cc:460] Webserver started at http://127.0.0.1:45907/ using document root <none> and password file <none>
I20260925 10:37:16.484716 20981 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:16.484858 20981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:16.485140 20981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:16.487957 20981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "020b5b29a13b41a5bc55a0d6738bb241"
format_stamp: "Formatted at 2026-09-25 10:37:16 on dist-test-slave-dxdd"
server_key: "ead6adc4c2d0dfc85ccdf74949cb98c3"
server_key_iv: "54a5356381f816362d492ca0e722120e"
server_key_version: "encryptionkey@0"
I20260925 10:37:16.488688 20981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "020b5b29a13b41a5bc55a0d6738bb241"
format_stamp: "Formatted at 2026-09-25 10:37:16 on dist-test-slave-dxdd"
server_key: "ead6adc4c2d0dfc85ccdf74949cb98c3"
server_key_iv: "54a5356381f816362d492ca0e722120e"
server_key_version: "encryptionkey@0"
I20260925 10:37:16.493599 20981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 10:37:16.496974 20996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:16.498196 20981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 10:37:16.498414 20981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "020b5b29a13b41a5bc55a0d6738bb241"
format_stamp: "Formatted at 2026-09-25 10:37:16 on dist-test-slave-dxdd"
server_key: "ead6adc4c2d0dfc85ccdf74949cb98c3"
server_key_iv: "54a5356381f816362d492ca0e722120e"
server_key_version: "encryptionkey@0"
I20260925 10:37:16.498610 20981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:16.504026 20981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:16.507920 20981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:16.508209 20981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:16.525854 20981 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34551
I20260925 10:37:16.525903 21047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34551 every 8 connection(s)
I20260925 10:37:16.527209 20981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:16.530718 21048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:16.537012 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 20981
I20260925 10:37:16.537190 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:16.537698 388 external_mini_cluster.cc:1428] Setting key c0fc87eee8faf5e276e7dd6363e1b2e9
I20260925 10:37:16.541121 21048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241: Bootstrap starting.
I20260925 10:37:16.544540 21048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:16.545089 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:16.539390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40467 (local address 127.0.0.1:34551)
0925 10:37:16.540156 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:16.540171 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:16.540211 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:37:16.541904 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:16.541926 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:16.542238 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:16.542620 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:16.543606 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:16.543612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:16.543622 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:16.543635 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:16.543689 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:16.543708 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:16.543858 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:16.543995 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 10:37:16.544184 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":359,"threads_started":1}
I20260925 10:37:16.545621 21048 log.cc:824] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:16.548579 21048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241: No bootstrap required, opened a new log
I20260925 10:37:16.548748 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:16.556315 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:16.553722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37111 (local address 127.0.0.1:34551)
0925 10:37:16.553907 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:16.553917 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:16.554404 (+ 487us) server_negotiation.cc:407] Connection header received
0925 10:37:16.554566 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:16.554571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:16.554624 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:16.554734 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:16.555562 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:16.555571 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:16.555576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:16.555592 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:16.555644 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:16.555663 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:16.555789 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:16.555915 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 10:37:16.555963 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:37:16.558524 21048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020b5b29a13b41a5bc55a0d6738bb241" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34551 } }
I20260925 10:37:16.558873 21048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:16.558972 21048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 020b5b29a13b41a5bc55a0d6738bb241, State: Initialized, Role: FOLLOWER
I20260925 10:37:16.559583 21048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020b5b29a13b41a5bc55a0d6738bb241" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34551 } }
I20260925 10:37:16.559897 21048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:16.560081 21048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:16.560261 21048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:16.562547 21048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020b5b29a13b41a5bc55a0d6738bb241" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34551 } }
I20260925 10:37:16.563073 21048 leader_election.cc:302] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 020b5b29a13b41a5bc55a0d6738bb241; no voters:
I20260925 10:37:16.563838 21048 leader_election.cc:288] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:16.563899 21058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:16.564679 21058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [term 1 LEADER]: Becoming Leader. State: Replica: 020b5b29a13b41a5bc55a0d6738bb241, State: Running, Role: LEADER
I20260925 10:37:16.565263 21058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020b5b29a13b41a5bc55a0d6738bb241" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34551 } }
I20260925 10:37:16.565971 21048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:16.567904 21059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "020b5b29a13b41a5bc55a0d6738bb241" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020b5b29a13b41a5bc55a0d6738bb241" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34551 } } }
I20260925 10:37:16.568392 21059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:16.567931 21060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 020b5b29a13b41a5bc55a0d6738bb241. Latest consensus state: current_term: 1 leader_uuid: "020b5b29a13b41a5bc55a0d6738bb241" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020b5b29a13b41a5bc55a0d6738bb241" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34551 } } }
I20260925 10:37:16.575026 21060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:16.576056 21065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:16.581506 21065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:16.590375 21065 catalog_manager.cc:1347] Generated new cluster ID: f5ccb8f16b574e97ace9f36bd59c3b74
I20260925 10:37:16.590479 21065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:16.626818 21065 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:16.627254 21065 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:16.636562 21065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 020b5b29a13b41a5bc55a0d6738bb241: Generated new TSK 0
I20260925 10:37:16.637424 21065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:16.647850 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 20981
W20260925 10:37:16.648751 21043 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40467
W20260925 10:37:16.649372 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 20981: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:37:16.649507 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 20981
2026-09-25T10:37:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1857 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:16 dist-test-slave-dxdd krb5kdc[21079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:16 dist-test-slave-dxdd krb5kdc[21079](info): set up 2 sockets
Sep 25 10:37:16 dist-test-slave-dxdd krb5kdc[21079](info): commencing operation
krb5kdc: starting...
W20260925 10:37:17.738528 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:17 dist-test-slave-dxdd krb5kdc[21079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:37:17.973549 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:18.289088 21092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:18.289384 21092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:18.289542 21092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:18.296519 21092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:18.296648 21092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:18.296736 21092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:18.296824 21092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:18.296905 21092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:18.306450 21092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:18.307696 21092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:18.308979 21092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:18.320525 21097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:18.320564 21100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:18.320590 21098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:18.321353 21092 server_base.cc:1034] running on GCE node
I20260925 10:37:18.322082 21092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:18.323484 21092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:18.324677 21092 hybrid_clock.cc:648] HybridClock initialized: now 1790332638324638 us; error 33 us; skew 500 ppm
I20260925 10:37:18.325026 21092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:18 dist-test-slave-dxdd krb5kdc[21079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:18.328214 21092 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:37:18.329632 21092 webserver.cc:460] Webserver started at http://127.0.0.1:45301/ using document root <none> and password file <none>
I20260925 10:37:18.330289 21092 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:18.330471 21092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:18.330770 21092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:18.333662 21092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "057cf4815ebe48169f5684ac297d3969"
format_stamp: "Formatted at 2026-09-25 10:37:18 on dist-test-slave-dxdd"
server_key: "85eed623e7bf84998217b06725db263a"
server_key_iv: "ee5cf7257f26271039b6b3ba7d6cb97d"
server_key_version: "encryptionkey@0"
I20260925 10:37:18.334494 21092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "057cf4815ebe48169f5684ac297d3969"
format_stamp: "Formatted at 2026-09-25 10:37:18 on dist-test-slave-dxdd"
server_key: "85eed623e7bf84998217b06725db263a"
server_key_iv: "ee5cf7257f26271039b6b3ba7d6cb97d"
server_key_version: "encryptionkey@0"
I20260925 10:37:18.339370 21092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 10:37:18.342609 21107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:18.343823 21092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:37:18.344015 21092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "057cf4815ebe48169f5684ac297d3969"
format_stamp: "Formatted at 2026-09-25 10:37:18 on dist-test-slave-dxdd"
server_key: "85eed623e7bf84998217b06725db263a"
server_key_iv: "ee5cf7257f26271039b6b3ba7d6cb97d"
server_key_version: "encryptionkey@0"
I20260925 10:37:18.344202 21092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:18.359289 21092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:18.362862 21092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:18.363138 21092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:18.380618 21092 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40267
I20260925 10:37:18.380648 21158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40267 every 8 connection(s)
I20260925 10:37:18.381932 21092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:18.385564 21159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:18.385946 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 21092
I20260925 10:37:18.386118 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:18.386592 388 external_mini_cluster.cc:1428] Setting key afc4fc09cd95aeb3a83d9a4d0ff10c10
I20260925 10:37:18.396845 21159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969: Bootstrap starting.
I20260925 10:37:18.399835 21159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969: Neither blocks nor log segments found. Creating new log.
Sep 25 10:37:18 dist-test-slave-dxdd krb5kdc[21079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:37:18.401074 21159 log.cc:824] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:18.403910 21159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969: No bootstrap required, opened a new log
I20260925 10:37:18.408107 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:18.388290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57383 (local address 127.0.0.1:40267)
0925 10:37:18.389002 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:18.389012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:18.389047 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:37:18.391311 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:18.391334 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:18.391696 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:18.392050 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:18.393498 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:18.394838 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:18.396060 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:18.396685 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:18.396870 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:18.401095 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:18.401102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:18.401114 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:18.401167 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:18.404210 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:18.405025 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:18.405030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:18.405035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:18.405182 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:18.405627 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:18.405630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:18.405632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:18.406112 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:18.406376 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:18.406854 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 10:37:18.407145 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":307,"threads_started":1}
I20260925 10:37:18.411793 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:18.413540 21159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057cf4815ebe48169f5684ac297d3969" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40267 } }
I20260925 10:37:18.413955 21159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:18.414059 21159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 057cf4815ebe48169f5684ac297d3969, State: Initialized, Role: FOLLOWER
I20260925 10:37:18.414681 21159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057cf4815ebe48169f5684ac297d3969" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40267 } }
I20260925 10:37:18.415009 21159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:18.415220 21159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:18.415400 21159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:18.417982 21159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057cf4815ebe48169f5684ac297d3969" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40267 } }
I20260925 10:37:18.418526 21159 leader_election.cc:302] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 057cf4815ebe48169f5684ac297d3969; no voters:
I20260925 10:37:18.419163 21159 leader_election.cc:288] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:18.419278 21169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:18.421056 21169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [term 1 LEADER]: Becoming Leader. State: Replica: 057cf4815ebe48169f5684ac297d3969, State: Running, Role: LEADER
I20260925 10:37:18.421554 21159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:18.421692 21169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057cf4815ebe48169f5684ac297d3969" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40267 } }
I20260925 10:37:18.431095 21170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "057cf4815ebe48169f5684ac297d3969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057cf4815ebe48169f5684ac297d3969" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40267 } } }
I20260925 10:37:18.432994 21170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:18.432677 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:18.416822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52587 (local address 127.0.0.1:40267)
0925 10:37:18.416997 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:18.417003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:18.418713 (+ 1710us) server_negotiation.cc:407] Connection header received
0925 10:37:18.418902 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:18.418907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:18.418978 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:18.419071 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:18.420785 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:18.421794 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:18.423248 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:18.423719 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:18.423850 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:18.425991 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:18.425997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:18.426003 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:18.426039 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:18.428904 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:18.429525 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:18.429531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:18.429534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:18.429676 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:18.430124 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:18.430128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:18.430135 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:18.430705 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:18.430961 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:18.432064 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0925 10:37:18.432283 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:37:18.432261 21171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 057cf4815ebe48169f5684ac297d3969. Latest consensus state: current_term: 1 leader_uuid: "057cf4815ebe48169f5684ac297d3969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057cf4815ebe48169f5684ac297d3969" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40267 } } }
I20260925 10:37:18.435329 21171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:18.436492 21179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:18.441365 21179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:18.449471 21179 catalog_manager.cc:1347] Generated new cluster ID: 113b5a99b7ae4039b11512ddd639b601
I20260925 10:37:18.449615 21179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:18.466156 21179 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:18.467384 21179 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:18.473554 21179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 057cf4815ebe48169f5684ac297d3969: Generated new TSK 0
I20260925 10:37:18.474282 21179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:18.483875 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 21092
I20260925 10:37:18.485128 21154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:18.572453 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 21092
2026-09-25T10:37:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1922 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:18 dist-test-slave-dxdd krb5kdc[21191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:18 dist-test-slave-dxdd krb5kdc[21191](info): set up 2 sockets
Sep 25 10:37:18 dist-test-slave-dxdd krb5kdc[21191](info): commencing operation
krb5kdc: starting...
W20260925 10:37:19.666623 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:19 dist-test-slave-dxdd krb5kdc[21191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:37:19.895514 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:20.208062 21204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:20.208372 21204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:20.208532 21204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:20.215372 21204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:20.215543 21204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:20.215663 21204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:20.215775 21204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:20.215848 21204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:20.225023 21204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:20.226156 21204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:20.227438 21204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:20.238607 21209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:20.238649 21212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:20.238564 21210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:20.239672 21204 server_base.cc:1034] running on GCE node
I20260925 10:37:20.240598 21204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:20.241909 21204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:20.243134 21204 hybrid_clock.cc:648] HybridClock initialized: now 1790332640243099 us; error 52 us; skew 500 ppm
I20260925 10:37:20.243491 21204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:20 dist-test-slave-dxdd krb5kdc[21191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:20.246587 21204 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:37:20.247951 21204 webserver.cc:460] Webserver started at http://127.0.0.1:45339/ using document root <none> and password file <none>
I20260925 10:37:20.248610 21204 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:20.248754 21204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:20.249049 21204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:20.251924 21204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "6a5d5cc67e1547ad8840a69b9f98453f"
format_stamp: "Formatted at 2026-09-25 10:37:20 on dist-test-slave-dxdd"
server_key: "7192c4d1632a5680ca3188a5e191840a"
server_key_iv: "7c4f0cb0ead6f6140917f0ea957a5425"
server_key_version: "encryptionkey@0"
I20260925 10:37:20.252640 21204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "6a5d5cc67e1547ad8840a69b9f98453f"
format_stamp: "Formatted at 2026-09-25 10:37:20 on dist-test-slave-dxdd"
server_key: "7192c4d1632a5680ca3188a5e191840a"
server_key_iv: "7c4f0cb0ead6f6140917f0ea957a5425"
server_key_version: "encryptionkey@0"
I20260925 10:37:20.257309 21204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 10:37:20.260481 21219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:20.261595 21204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:37:20.261768 21204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "6a5d5cc67e1547ad8840a69b9f98453f"
format_stamp: "Formatted at 2026-09-25 10:37:20 on dist-test-slave-dxdd"
server_key: "7192c4d1632a5680ca3188a5e191840a"
server_key_iv: "7c4f0cb0ead6f6140917f0ea957a5425"
server_key_version: "encryptionkey@0"
I20260925 10:37:20.261946 21204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:20.275810 21204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:20.279853 21204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:20.280112 21204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:20.297515 21204 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38419
I20260925 10:37:20.297595 21270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38419 every 8 connection(s)
I20260925 10:37:20.298799 21204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:20.302428 21271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:20.305711 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 21204
I20260925 10:37:20.305920 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:20.306542 388 external_mini_cluster.cc:1428] Setting key 5bb8eefb49007caae01ba28fcbbbae20
I20260925 10:37:20.315163 21271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f: Bootstrap starting.
I20260925 10:37:20.318670 21271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:20.319816 21271 log.cc:824] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:20.320151 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:20.308582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54089 (local address 127.0.0.1:38419)
0925 10:37:20.309458 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:20.309470 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:20.309510 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:37:20.311358 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:20.311382 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:20.311787 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:20.312146 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:20.312733 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:20.313993 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:20.315880 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:20.316527 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:20.316677 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:20.317888 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:20.317895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:20.317907 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:20.318460 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:20.318543 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:20.318569 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:20.318659 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:20.318904 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 10:37:20.319218 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":407,"threads_started":1}
I20260925 10:37:20.322495 21271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f: No bootstrap required, opened a new log
I20260925 10:37:20.323558 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:20.330523 21271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5d5cc67e1547ad8840a69b9f98453f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38419 } }
I20260925 10:37:20.330868 21271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:20.330960 21271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a5d5cc67e1547ad8840a69b9f98453f, State: Initialized, Role: FOLLOWER
I20260925 10:37:20.331566 21271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5d5cc67e1547ad8840a69b9f98453f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38419 } }
I20260925 10:37:20.331871 21271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:20.332015 21271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:20.332197 21271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:20.334064 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:20.327625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37393 (local address 127.0.0.1:38419)
0925 10:37:20.327811 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:20.327817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:20.328153 (+ 336us) server_negotiation.cc:407] Connection header received
0925 10:37:20.328301 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:20.328307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:20.328385 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:20.328487 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:20.328932 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:20.329980 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:20.331465 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:20.332196 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:20.332332 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:20.333350 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:20.333356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:20.333362 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:20.333382 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:20.333441 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:20.333462 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:20.333561 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:20.333657 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 10:37:20.333746 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:37:20.335057 21271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5d5cc67e1547ad8840a69b9f98453f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38419 } }
I20260925 10:37:20.335561 21271 leader_election.cc:302] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a5d5cc67e1547ad8840a69b9f98453f; no voters:
I20260925 10:37:20.336074 21271 leader_election.cc:288] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:20.336215 21281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:20.336889 21281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [term 1 LEADER]: Becoming Leader. State: Replica: 6a5d5cc67e1547ad8840a69b9f98453f, State: Running, Role: LEADER
I20260925 10:37:20.337430 21281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5d5cc67e1547ad8840a69b9f98453f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38419 } }
I20260925 10:37:20.338008 21271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:20.340310 21283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a5d5cc67e1547ad8840a69b9f98453f. Latest consensus state: current_term: 1 leader_uuid: "6a5d5cc67e1547ad8840a69b9f98453f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5d5cc67e1547ad8840a69b9f98453f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38419 } } }
I20260925 10:37:20.340999 21283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:20.340912 21282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a5d5cc67e1547ad8840a69b9f98453f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5d5cc67e1547ad8840a69b9f98453f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38419 } } }
I20260925 10:37:20.341287 21282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:20.343993 21289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:20.349987 21289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:20.357710 21289 catalog_manager.cc:1347] Generated new cluster ID: e5738d2b7cae4a9e8e557b46762a5bff
I20260925 10:37:20.357822 21289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:20.375347 21289 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:20.376504 21289 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:20.382071 21289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a5d5cc67e1547ad8840a69b9f98453f: Generated new TSK 0
I20260925 10:37:20.382870 21289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:20.390863 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 21204
W20260925 10:37:20.392012 21266 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54089
W20260925 10:37:20.392630 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 21204: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:37:20.392776 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 21204
2026-09-25T10:37:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1819 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:20 dist-test-slave-dxdd krb5kdc[21302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:20 dist-test-slave-dxdd krb5kdc[21302](info): set up 2 sockets
Sep 25 10:37:20 dist-test-slave-dxdd krb5kdc[21302](info): commencing operation
krb5kdc: starting...
W20260925 10:37:21.474624 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:21 dist-test-slave-dxdd krb5kdc[21302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:37:21.705386 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:22.018935 21315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:22.019228 21315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:22.019425 21315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:22.026484 21315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:22.026602 21315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:22.026703 21315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:22.026818 21315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:22.026899 21315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:22.036284 21315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:22.037442 21315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:22.038699 21315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:22.049513 21320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:22.049568 21321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:22.049597 21323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:22.050614 21315 server_base.cc:1034] running on GCE node
I20260925 10:37:22.051352 21315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:22.052681 21315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:22.053907 21315 hybrid_clock.cc:648] HybridClock initialized: now 1790332642053876 us; error 43 us; skew 500 ppm
I20260925 10:37:22.054229 21315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:22 dist-test-slave-dxdd krb5kdc[21302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:22.063228 21315 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:37:22.064675 21315 webserver.cc:460] Webserver started at http://127.0.0.1:46167/ using document root <none> and password file <none>
I20260925 10:37:22.065312 21315 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:22.065448 21315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:22.065734 21315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:22.068517 21315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "cf97183fe905402d93fd8edd2cd7a4c3"
format_stamp: "Formatted at 2026-09-25 10:37:22 on dist-test-slave-dxdd"
server_key: "83d57494513daac1b95fa0719d5421d6"
server_key_iv: "4da764891a1a70ed24c459866d4dc2f3"
server_key_version: "encryptionkey@0"
I20260925 10:37:22.069232 21315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "cf97183fe905402d93fd8edd2cd7a4c3"
format_stamp: "Formatted at 2026-09-25 10:37:22 on dist-test-slave-dxdd"
server_key: "83d57494513daac1b95fa0719d5421d6"
server_key_iv: "4da764891a1a70ed24c459866d4dc2f3"
server_key_version: "encryptionkey@0"
I20260925 10:37:22.073968 21315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260925 10:37:22.077183 21330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:22.078455 21315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:37:22.078619 21315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "cf97183fe905402d93fd8edd2cd7a4c3"
format_stamp: "Formatted at 2026-09-25 10:37:22 on dist-test-slave-dxdd"
server_key: "83d57494513daac1b95fa0719d5421d6"
server_key_iv: "4da764891a1a70ed24c459866d4dc2f3"
server_key_version: "encryptionkey@0"
I20260925 10:37:22.078799 21315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:22.084209 21315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:22.087834 21315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:22.088128 21315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:22.105684 21315 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46807
I20260925 10:37:22.105723 21381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46807 every 8 connection(s)
I20260925 10:37:22.107082 21315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:22.110563 21382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:22.115803 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 21315
I20260925 10:37:22.115980 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:22.116377 388 external_mini_cluster.cc:1428] Setting key a9ff5ebe7b1780eb93758a5bb77e0bfc
I20260925 10:37:22.120733 21382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3: Bootstrap starting.
I20260925 10:37:22.123827 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:22.118017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37465 (local address 127.0.0.1:46807)
0925 10:37:22.118854 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:22.118868 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:22.118908 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:37:22.120526 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:22.120547 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:22.120912 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:22.121246 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:22.122287 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:22.122296 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:22.122350 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:22.122374 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:22.122434 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:22.122457 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:22.122620 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:22.122758 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 10:37:22.122935 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":399,"threads_started":1}
I20260925 10:37:22.124451 21382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:22.125589 21382 log.cc:824] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:22.127458 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:22.128556 21382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3: No bootstrap required, opened a new log
I20260925 10:37:22.134608 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:22.132210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41759 (local address 127.0.0.1:46807)
0925 10:37:22.132391 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:22.132397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:22.132863 (+ 466us) server_negotiation.cc:407] Connection header received
0925 10:37:22.132988 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:22.132992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:22.133046 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:22.133147 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:22.133987 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:22.133993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:22.133998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:22.134015 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:22.134065 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:22.134085 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:22.134205 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:22.134251 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 10:37:22.134334 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:37:22.137430 21382 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf97183fe905402d93fd8edd2cd7a4c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } }
I20260925 10:37:22.137760 21382 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:22.137867 21382 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf97183fe905402d93fd8edd2cd7a4c3, State: Initialized, Role: FOLLOWER
I20260925 10:37:22.138491 21382 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf97183fe905402d93fd8edd2cd7a4c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } }
I20260925 10:37:22.138720 21382 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:22.138859 21382 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:22.139026 21382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:22.141153 21382 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf97183fe905402d93fd8edd2cd7a4c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } }
I20260925 10:37:22.141637 21382 leader_election.cc:302] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf97183fe905402d93fd8edd2cd7a4c3; no voters:
I20260925 10:37:22.142162 21382 leader_election.cc:288] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:22.142287 21392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:22.143000 21392 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [term 1 LEADER]: Becoming Leader. State: Replica: cf97183fe905402d93fd8edd2cd7a4c3, State: Running, Role: LEADER
I20260925 10:37:22.143464 21392 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf97183fe905402d93fd8edd2cd7a4c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } }
I20260925 10:37:22.144229 21382 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:22.146482 21394 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf97183fe905402d93fd8edd2cd7a4c3. Latest consensus state: current_term: 1 leader_uuid: "cf97183fe905402d93fd8edd2cd7a4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf97183fe905402d93fd8edd2cd7a4c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } } }
I20260925 10:37:22.146456 21393 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf97183fe905402d93fd8edd2cd7a4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf97183fe905402d93fd8edd2cd7a4c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } } }
I20260925 10:37:22.146940 21394 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:22.146940 21393 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:22.148552 21398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:22.153047 21398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:22.161712 21398 catalog_manager.cc:1347] Generated new cluster ID: 89052d27b23e4a9d84e66b684106ac31
I20260925 10:37:22.161840 21398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:22.182533 21398 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:22.182979 21398 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:22.190397 21398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf97183fe905402d93fd8edd2cd7a4c3: Generated new TSK 0
I20260925 10:37:22.191099 21398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:22.201807 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 21315
W20260925 10:37:22.203192 21377 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37465
W20260925 10:37:22.203866 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 21315: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:37:22.204013 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 21315
2026-09-25T10:37:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1812 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:22 dist-test-slave-dxdd krb5kdc[21413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:22 dist-test-slave-dxdd krb5kdc[21413](info): set up 2 sockets
Sep 25 10:37:22 dist-test-slave-dxdd krb5kdc[21413](info): commencing operation
krb5kdc: starting...
W20260925 10:37:23.290616 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:23 dist-test-slave-dxdd krb5kdc[21413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:37:23.527146 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42223
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:23.856812 21426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:23.857129 21426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:23.857295 21426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:23.864323 21426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:23.864416 21426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:23.864542 21426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:23.864650 21426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:23.864728 21426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:23.874084 21426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:23.875344 21426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:23.876648 21426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:23.887939 21431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:23.888017 21434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:23.888041 21432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:23.889106 21426 server_base.cc:1034] running on GCE node
I20260925 10:37:23.889900 21426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:23.891301 21426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:23.892541 21426 hybrid_clock.cc:648] HybridClock initialized: now 1790332643892502 us; error 48 us; skew 500 ppm
I20260925 10:37:23.892905 21426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:23 dist-test-slave-dxdd krb5kdc[21413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:23.896075 21426 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:37:23.897558 21426 webserver.cc:460] Webserver started at http://127.0.0.1:43823/ using document root <none> and password file <none>
I20260925 10:37:23.898217 21426 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:23.898414 21426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:23.898728 21426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:23.902163 21426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "7a30a66fcbe04ae7a0986febc89911e1"
format_stamp: "Formatted at 2026-09-25 10:37:23 on dist-test-slave-dxdd"
server_key: "06502e145b121b258286eb25cc4eedad"
server_key_iv: "96b46601cc6545d212c8379444d4a1f0"
server_key_version: "encryptionkey@0"
I20260925 10:37:23.903004 21426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "7a30a66fcbe04ae7a0986febc89911e1"
format_stamp: "Formatted at 2026-09-25 10:37:23 on dist-test-slave-dxdd"
server_key: "06502e145b121b258286eb25cc4eedad"
server_key_iv: "96b46601cc6545d212c8379444d4a1f0"
server_key_version: "encryptionkey@0"
I20260925 10:37:23.907995 21426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 10:37:23.911172 21441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:23.912448 21426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:37:23.912652 21426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "7a30a66fcbe04ae7a0986febc89911e1"
format_stamp: "Formatted at 2026-09-25 10:37:23 on dist-test-slave-dxdd"
server_key: "06502e145b121b258286eb25cc4eedad"
server_key_iv: "96b46601cc6545d212c8379444d4a1f0"
server_key_version: "encryptionkey@0"
I20260925 10:37:23.912843 21426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:23.925505 21426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:23.929567 21426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:23.929841 21426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:23.947750 21426 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42223
I20260925 10:37:23.947781 21492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42223 every 8 connection(s)
I20260925 10:37:23.949021 21426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:23.952375 21493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:23.959096 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 21426
I20260925 10:37:23.959287 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:23.959678 388 external_mini_cluster.cc:1428] Setting key 2c7a043e7138310fa8acc10fe664c787
I20260925 10:37:23.962232 21493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1: Bootstrap starting.
I20260925 10:37:23.965415 21493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:23.966655 21493 log.cc:824] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:23.969673 21493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1: No bootstrap required, opened a new log
Sep 25 10:37:23 dist-test-slave-dxdd krb5kdc[21413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:37:23.978199 21493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a30a66fcbe04ae7a0986febc89911e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } }
I20260925 10:37:23.978641 21493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:23.978749 21493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a30a66fcbe04ae7a0986febc89911e1, State: Initialized, Role: FOLLOWER
I20260925 10:37:23.979329 21493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a30a66fcbe04ae7a0986febc89911e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } }
I20260925 10:37:23.979584 21493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:23.979738 21493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:23.979915 21493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:23.982441 21493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a30a66fcbe04ae7a0986febc89911e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } }
I20260925 10:37:23.982911 21493 leader_election.cc:302] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a30a66fcbe04ae7a0986febc89911e1; no voters:
I20260925 10:37:23.983011 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:23.961408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:55411 (local address 127.0.0.1:42223)
0925 10:37:23.962119 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:23.962130 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:23.962166 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:37:23.963912 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:23.963938 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:23.964281 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:23.964580 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:23.966423 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:23.968117 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:23.969912 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:23.970638 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:23.970758 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:23.975408 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:23.975415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:23.975432 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:23.975464 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:23.978513 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:23.979488 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:23.979494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:23.979499 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:23.979722 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:23.980216 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:23.980221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:23.980224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:23.980737 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:23.981087 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:23.981544 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 10:37:23.981847 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":321,"threads_started":1}
I20260925 10:37:23.983579 21493 leader_election.cc:288] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:23.983690 21498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:23.985610 21493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:23.985782 21498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [term 1 LEADER]: Becoming Leader. State: Replica: 7a30a66fcbe04ae7a0986febc89911e1, State: Running, Role: LEADER
I20260925 10:37:23.986425 21498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a30a66fcbe04ae7a0986febc89911e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } }
I20260925 10:37:23.988109 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:23.992600 21499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a30a66fcbe04ae7a0986febc89911e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a30a66fcbe04ae7a0986febc89911e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } } }
I20260925 10:37:23.993041 21499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:23.993613 21500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a30a66fcbe04ae7a0986febc89911e1. Latest consensus state: current_term: 1 leader_uuid: "7a30a66fcbe04ae7a0986febc89911e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a30a66fcbe04ae7a0986febc89911e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } } }
I20260925 10:37:23.993918 21500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:24.001740 21509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:24.007812 21509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:24.020113 21521 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:37:24.021862 21509 catalog_manager.cc:1347] Generated new cluster ID: 027942a94f9d4486a38d4de04fc7c257
I20260925 10:37:24.022024 21509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:24.023779 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:24.005017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:46495 (local address 127.0.0.1:42223)
0925 10:37:24.005200 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:24.005206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:24.006918 (+ 1712us) server_negotiation.cc:407] Connection header received
0925 10:37:24.007136 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:24.007141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:24.007215 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:24.007314 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:24.009235 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:24.010434 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:24.012309 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:24.013009 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:24.013133 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:24.016473 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:24.016484 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:24.016489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:24.016524 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:24.020254 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:24.021484 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:24.021489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:24.021496 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:24.021654 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:24.022299 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:24.022328 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:24.022333 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:24.022660 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:24.022959 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:24.023329 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 10:37:24.023501 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:37:24.038610 21509 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:24.039806 21509 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:24.047161 21509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a30a66fcbe04ae7a0986febc89911e1: Generated new TSK 0
I20260925 10:37:24.047921 21509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:24.055790 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 21426
I20260925 10:37:24.056838 21488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:24.146579 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 21426
W20260925 10:37:24.163463 21411 connection.cc:535] client connection to 127.0.0.1:42223 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42223): Connection reset by peer (error 104)
2026-09-25T10:37:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:24 dist-test-slave-dxdd krb5kdc[21526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:24 dist-test-slave-dxdd krb5kdc[21526](info): set up 2 sockets
Sep 25 10:37:24 dist-test-slave-dxdd krb5kdc[21526](info): commencing operation
krb5kdc: starting...
W20260925 10:37:25.238714 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:25 dist-test-slave-dxdd krb5kdc[21526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:37:25.479022 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41873
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:25.797683 21539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:25.797941 21539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:25.798120 21539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:25.805285 21539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:25.805397 21539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:25.805444 21539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:25.805531 21539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:25.805634 21539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:25.815114 21539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:25.816344 21539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:25.817617 21539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:25.829427 21547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:25.829425 21545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:25.829487 21544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:25.830163 21539 server_base.cc:1034] running on GCE node
I20260925 10:37:25.830922 21539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:25.832222 21539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:25.833413 21539 hybrid_clock.cc:648] HybridClock initialized: now 1790332645833356 us; error 49 us; skew 500 ppm
I20260925 10:37:25.833765 21539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:25 dist-test-slave-dxdd krb5kdc[21526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:25.836840 21539 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:37:25.838222 21539 webserver.cc:460] Webserver started at http://127.0.0.1:36357/ using document root <none> and password file <none>
I20260925 10:37:25.838922 21539 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:25.839051 21539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:25.839355 21539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:25.842199 21539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "ecc9ba8dfb194a3bad8b14ac5be000fb"
format_stamp: "Formatted at 2026-09-25 10:37:25 on dist-test-slave-dxdd"
server_key: "f5dd6b8535c49c7a19476db53f0f9e0a"
server_key_iv: "bf59a63d9b205228e1aad6fc6c226e51"
server_key_version: "encryptionkey@0"
I20260925 10:37:25.843000 21539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "ecc9ba8dfb194a3bad8b14ac5be000fb"
format_stamp: "Formatted at 2026-09-25 10:37:25 on dist-test-slave-dxdd"
server_key: "f5dd6b8535c49c7a19476db53f0f9e0a"
server_key_iv: "bf59a63d9b205228e1aad6fc6c226e51"
server_key_version: "encryptionkey@0"
I20260925 10:37:25.847815 21539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:37:25.851039 21554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:25.852344 21539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:37:25.852514 21539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "ecc9ba8dfb194a3bad8b14ac5be000fb"
format_stamp: "Formatted at 2026-09-25 10:37:25 on dist-test-slave-dxdd"
server_key: "f5dd6b8535c49c7a19476db53f0f9e0a"
server_key_iv: "bf59a63d9b205228e1aad6fc6c226e51"
server_key_version: "encryptionkey@0"
I20260925 10:37:25.852701 21539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:25.871793 21539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:25.875574 21539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:25.875847 21539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:25.893772 21539 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41873
I20260925 10:37:25.893810 21605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41873 every 8 connection(s)
I20260925 10:37:25.895141 21539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:25.898749 21606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:25.902508 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 21539
I20260925 10:37:25.902688 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:25.903200 388 external_mini_cluster.cc:1428] Setting key dff741af1feeb650336d479f1525b420
I20260925 10:37:25.911121 21606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb: Bootstrap starting.
I20260925 10:37:25.914669 21606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:25.915720 21606 log.cc:824] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:25.916049 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:25.904891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:42527 (local address 127.0.0.1:41873)
0925 10:37:25.905616 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:25.905628 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:25.905667 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:37:25.907318 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:25.907340 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:25.907750 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:25.908075 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:25.908616 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:25.909897 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:25.911700 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:25.912334 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:25.912468 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:25.913769 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:25.913776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:25.913792 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:25.914294 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:25.914426 (+ 132us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:25.914450 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:25.914573 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:25.914915 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 10:37:25.915118 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":339,"threads_started":1}
I20260925 10:37:25.918448 21606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb: No bootstrap required, opened a new log
I20260925 10:37:25.919656 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:25.928398 21606 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc9ba8dfb194a3bad8b14ac5be000fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } }
I20260925 10:37:25.928855 21606 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:25.928970 21606 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc9ba8dfb194a3bad8b14ac5be000fb, State: Initialized, Role: FOLLOWER
I20260925 10:37:25.929632 21606 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc9ba8dfb194a3bad8b14ac5be000fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } }
I20260925 10:37:25.929899 21606 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:25.930044 21606 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:25.930248 21606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:25.931267 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:25.924617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:48665 (local address 127.0.0.1:41873)
0925 10:37:25.924808 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:25.924814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:25.925289 (+ 475us) server_negotiation.cc:407] Connection header received
0925 10:37:25.925461 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:25.925466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:25.925546 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:25.925655 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:25.926102 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:25.927336 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:25.928597 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:25.929256 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:25.929381 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:25.930515 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:25.930521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:25.930525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:25.930552 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:25.930606 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:25.930622 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:25.930741 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:25.930932 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 10:37:25.930984 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:37:25.932966 21606 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc9ba8dfb194a3bad8b14ac5be000fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } }
I20260925 10:37:25.933454 21606 leader_election.cc:302] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecc9ba8dfb194a3bad8b14ac5be000fb; no voters:
I20260925 10:37:25.933984 21606 leader_election.cc:288] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:25.934105 21616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:25.934818 21616 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [term 1 LEADER]: Becoming Leader. State: Replica: ecc9ba8dfb194a3bad8b14ac5be000fb, State: Running, Role: LEADER
I20260925 10:37:25.935367 21616 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc9ba8dfb194a3bad8b14ac5be000fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } }
I20260925 10:37:25.936075 21606 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:25.938161 21617 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecc9ba8dfb194a3bad8b14ac5be000fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc9ba8dfb194a3bad8b14ac5be000fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } } }
I20260925 10:37:25.938535 21617 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:25.938737 21618 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecc9ba8dfb194a3bad8b14ac5be000fb. Latest consensus state: current_term: 1 leader_uuid: "ecc9ba8dfb194a3bad8b14ac5be000fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc9ba8dfb194a3bad8b14ac5be000fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } } }
I20260925 10:37:25.939101 21618 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:25.942992 21622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:25.947901 21622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:25.956372 21622 catalog_manager.cc:1347] Generated new cluster ID: a4f07e0658d8481bbc401298e6c3e965
I20260925 10:37:25.956489 21622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:25.976022 21622 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:25.977983 21622 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:25.994565 21622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecc9ba8dfb194a3bad8b14ac5be000fb: Generated new TSK 0
I20260925 10:37:25.995662 21622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:26.002041 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 21539
W20260925 10:37:26.003250 21601 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.183:42527
W20260925 10:37:26.003984 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 21539: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:37:26.004137 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 21539
W20260925 10:37:26.021000 21524 connection.cc:535] client connection to 127.0.0.1:41873 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41873): Connection reset by peer (error 104)
2026-09-25T10:37:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1857 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:26 dist-test-slave-dxdd krb5kdc[21637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:26 dist-test-slave-dxdd krb5kdc[21637](info): set up 2 sockets
Sep 25 10:37:26 dist-test-slave-dxdd krb5kdc[21637](info): commencing operation
krb5kdc: starting...
W20260925 10:37:27.102871 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:27 dist-test-slave-dxdd krb5kdc[21637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:37:27.345618 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:27.665825 21650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:27.666119 21650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:27.666210 21650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:27.673314 21650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:27.673411 21650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:27.673538 21650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:27.673648 21650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:27.673738 21650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:27.683148 21650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:27.684383 21650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:27.685688 21650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:27.697813 21655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:27.697853 21658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:27.697877 21656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:27.698504 21650 server_base.cc:1034] running on GCE node
I20260925 10:37:27.699316 21650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:27.700722 21650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:27.701953 21650 hybrid_clock.cc:648] HybridClock initialized: now 1790332647701913 us; error 58 us; skew 500 ppm
I20260925 10:37:27.702358 21650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:27 dist-test-slave-dxdd krb5kdc[21637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:27.709287 21650 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:37:27.710808 21650 webserver.cc:460] Webserver started at http://127.0.0.1:46421/ using document root <none> and password file <none>
I20260925 10:37:27.711493 21650 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:27.711643 21650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:27.711954 21650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:27.714943 21650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "5e5215fb64614bafa6c7a2caf05f781d"
format_stamp: "Formatted at 2026-09-25 10:37:27 on dist-test-slave-dxdd"
server_key: "f36a882de778ca65355e8a3d52029a36"
server_key_iv: "ab032bd894bbefb1e9487d49d4d68dce"
server_key_version: "encryptionkey@0"
I20260925 10:37:27.715689 21650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "5e5215fb64614bafa6c7a2caf05f781d"
format_stamp: "Formatted at 2026-09-25 10:37:27 on dist-test-slave-dxdd"
server_key: "f36a882de778ca65355e8a3d52029a36"
server_key_iv: "ab032bd894bbefb1e9487d49d4d68dce"
server_key_version: "encryptionkey@0"
I20260925 10:37:27.720772 21650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 10:37:27.724107 21665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:27.725348 21650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:37:27.725526 21650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "5e5215fb64614bafa6c7a2caf05f781d"
format_stamp: "Formatted at 2026-09-25 10:37:27 on dist-test-slave-dxdd"
server_key: "f36a882de778ca65355e8a3d52029a36"
server_key_iv: "ab032bd894bbefb1e9487d49d4d68dce"
server_key_version: "encryptionkey@0"
I20260925 10:37:27.725720 21650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:27.731395 21650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:27.735635 21650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:27.735924 21650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:27.753948 21650 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33437
I20260925 10:37:27.753986 21716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33437 every 8 connection(s)
I20260925 10:37:27.755410 21650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:27.758945 21717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:27.759651 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 21650
I20260925 10:37:27.759826 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:27.760250 388 external_mini_cluster.cc:1428] Setting key d940a207cd52e04f1f74a0177828b01c
I20260925 10:37:27.769948 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:27.762103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:34493 (local address 127.0.0.1:33437)
0925 10:37:27.763171 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:27.763187 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:27.763258 (+ 71us) server_negotiation.cc:407] Connection header received
0925 10:37:27.766386 (+ 3128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:27.766419 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:27.766790 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:27.767212 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:27.768285 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:27.768292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:27.768305 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:27.768318 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:27.768376 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:27.768399 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:27.768662 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:27.768795 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 10:37:27.768987 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":479,"threads_started":1}
I20260925 10:37:27.773280 21717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d: Bootstrap starting.
I20260925 10:37:27.773797 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:27.777000 21717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:27.778076 21717 log.cc:824] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:27.780956 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:27.778260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:55703 (local address 127.0.0.1:33437)
0925 10:37:27.778509 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:27.778517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:27.778773 (+ 256us) server_negotiation.cc:407] Connection header received
0925 10:37:27.779066 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:27.779070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:27.779122 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:27.779220 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:27.780251 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:27.780256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:27.780261 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:27.780278 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:27.780324 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:27.780349 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:27.780477 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:27.780585 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 10:37:27.780648 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260925 10:37:27.781762 21717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d: No bootstrap required, opened a new log
I20260925 10:37:27.789106 21717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5215fb64614bafa6c7a2caf05f781d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33437 } }
I20260925 10:37:27.789453 21717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:27.789521 21717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5215fb64614bafa6c7a2caf05f781d, State: Initialized, Role: FOLLOWER
I20260925 10:37:27.789995 21717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5215fb64614bafa6c7a2caf05f781d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33437 } }
I20260925 10:37:27.790210 21717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:27.790380 21717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:27.790506 21717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:27.792871 21717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5215fb64614bafa6c7a2caf05f781d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33437 } }
I20260925 10:37:27.793334 21717 leader_election.cc:302] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e5215fb64614bafa6c7a2caf05f781d; no voters:
I20260925 10:37:27.793905 21717 leader_election.cc:288] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:27.794060 21727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:27.794757 21727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [term 1 LEADER]: Becoming Leader. State: Replica: 5e5215fb64614bafa6c7a2caf05f781d, State: Running, Role: LEADER
I20260925 10:37:27.795300 21727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5215fb64614bafa6c7a2caf05f781d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33437 } }
I20260925 10:37:27.796020 21717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:27.798027 21728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e5215fb64614bafa6c7a2caf05f781d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5215fb64614bafa6c7a2caf05f781d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33437 } } }
I20260925 10:37:27.798388 21728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:27.798895 21729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e5215fb64614bafa6c7a2caf05f781d. Latest consensus state: current_term: 1 leader_uuid: "5e5215fb64614bafa6c7a2caf05f781d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5215fb64614bafa6c7a2caf05f781d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33437 } } }
I20260925 10:37:27.799203 21729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:27.799290 21733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:27.804447 21733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:27.816579 21733 catalog_manager.cc:1347] Generated new cluster ID: 3559ca2ae4004e93b28988dcd1f97995
I20260925 10:37:27.816710 21733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:27.828732 21733 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:27.829245 21733 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:27.851205 21733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e5215fb64614bafa6c7a2caf05f781d: Generated new TSK 0
I20260925 10:37:27.852072 21733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:27.862864 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 21650
W20260925 10:37:27.863920 21712 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.183:34493
W20260925 10:37:27.864622 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 21650: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:37:27.864775 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 21650
2026-09-25T10:37:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1857 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:27 dist-test-slave-dxdd krb5kdc[21748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:27 dist-test-slave-dxdd krb5kdc[21748](info): set up 2 sockets
Sep 25 10:37:27 dist-test-slave-dxdd krb5kdc[21748](info): commencing operation
krb5kdc: starting...
W20260925 10:37:28.954792 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:29 dist-test-slave-dxdd krb5kdc[21748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:37:29.197458 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:29.510042 21761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:29.510293 21761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:29.510500 21761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:29.517464 21761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:29.517557 21761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:29.517602 21761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:29.517694 21761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:29.517755 21761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:29.527737 21761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43235
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:29.528966 21761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:29.530236 21761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:29.541962 21766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:29.542100 21769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:29.541986 21767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:29.542929 21761 server_base.cc:1034] running on GCE node
I20260925 10:37:29.543619 21761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:29.544938 21761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:29.546159 21761 hybrid_clock.cc:648] HybridClock initialized: now 1790332649546124 us; error 45 us; skew 500 ppm
I20260925 10:37:29.546540 21761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:29 dist-test-slave-dxdd krb5kdc[21748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:29.549554 21761 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:37:29.550940 21761 webserver.cc:460] Webserver started at http://127.0.0.1:38571/ using document root <none> and password file <none>
I20260925 10:37:29.551578 21761 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:29.551724 21761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:29.552021 21761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:29.554868 21761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "f030e61a060441aea4d87eb336b799cc"
format_stamp: "Formatted at 2026-09-25 10:37:29 on dist-test-slave-dxdd"
server_key: "b1625792afffaf93f63aa996558c4d9a"
server_key_iv: "a43c84a2b99e7eb8f8387c7aa658193f"
server_key_version: "encryptionkey@0"
I20260925 10:37:29.555600 21761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "f030e61a060441aea4d87eb336b799cc"
format_stamp: "Formatted at 2026-09-25 10:37:29 on dist-test-slave-dxdd"
server_key: "b1625792afffaf93f63aa996558c4d9a"
server_key_iv: "a43c84a2b99e7eb8f8387c7aa658193f"
server_key_version: "encryptionkey@0"
I20260925 10:37:29.560443 21761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:37:29.563673 21776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:29.564895 21761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:37:29.565069 21761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "f030e61a060441aea4d87eb336b799cc"
format_stamp: "Formatted at 2026-09-25 10:37:29 on dist-test-slave-dxdd"
server_key: "b1625792afffaf93f63aa996558c4d9a"
server_key_iv: "a43c84a2b99e7eb8f8387c7aa658193f"
server_key_version: "encryptionkey@0"
I20260925 10:37:29.565265 21761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:29.579672 21761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:29.583902 21761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:29.584226 21761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:29.601853 21761 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43235
I20260925 10:37:29.601889 21827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43235 every 8 connection(s)
I20260925 10:37:29.603269 21761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:29.606878 21828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:29.609390 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 21761
I20260925 10:37:29.609570 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:29.609974 388 external_mini_cluster.cc:1428] Setting key 9b487db885d585b9dc1083bc7fa667b0
I20260925 10:37:29.619320 21828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc: Bootstrap starting.
I20260925 10:37:29.622644 21828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:29.623564 21828 log.cc:824] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:37:29 dist-test-slave-dxdd krb5kdc[21748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:37:29.626425 21828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc: No bootstrap required, opened a new log
I20260925 10:37:29.632880 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:29.611781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:49721 (local address 127.0.0.1:43235)
0925 10:37:29.612575 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:29.612592 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:29.612637 (+ 45us) server_negotiation.cc:407] Connection header received
0925 10:37:29.614931 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:29.614960 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:29.615451 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:29.615893 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:29.617042 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:29.618986 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:29.620251 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:29.621326 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:29.621486 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:29.625719 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:29.625726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:29.625738 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:29.625773 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:29.628797 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:29.629656 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:29.629662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:29.629667 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:29.629836 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:29.630288 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:29.630291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:29.630293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:29.630838 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:29.631063 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:29.631601 (+ 538us) server_negotiation.cc:299] Negotiation successful
0925 10:37:29.631919 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":345,"threads_started":1}
I20260925 10:37:29.635005 21828 raft_consensus.cc:348] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f030e61a060441aea4d87eb336b799cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43235 } }
I20260925 10:37:29.635427 21828 raft_consensus.cc:374] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:29.635524 21828 raft_consensus.cc:729] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f030e61a060441aea4d87eb336b799cc, State: Initialized, Role: FOLLOWER
I20260925 10:37:29.636149 21828 consensus_queue.cc:260] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f030e61a060441aea4d87eb336b799cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43235 } }
I20260925 10:37:29.636422 21828 raft_consensus.cc:388] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:29.636580 21828 raft_consensus.cc:482] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:29.636587 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:29.636754 21828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:29.638913 21828 raft_consensus.cc:504] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f030e61a060441aea4d87eb336b799cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43235 } }
I20260925 10:37:29.639379 21828 leader_election.cc:302] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f030e61a060441aea4d87eb336b799cc; no voters:
I20260925 10:37:29.640076 21828 leader_election.cc:288] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:29.640200 21837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:29.641996 21837 raft_consensus.cc:686] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [term 1 LEADER]: Becoming Leader. State: Replica: f030e61a060441aea4d87eb336b799cc, State: Running, Role: LEADER
I20260925 10:37:29.642702 21837 consensus_queue.cc:237] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f030e61a060441aea4d87eb336b799cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43235 } }
I20260925 10:37:29.644840 21837 sys_catalog.cc:455] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f030e61a060441aea4d87eb336b799cc. Latest consensus state: current_term: 1 leader_uuid: "f030e61a060441aea4d87eb336b799cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f030e61a060441aea4d87eb336b799cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43235 } } }
I20260925 10:37:29.645153 21837 sys_catalog.cc:458] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:29.645818 21839 sys_catalog.cc:455] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f030e61a060441aea4d87eb336b799cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f030e61a060441aea4d87eb336b799cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43235 } } }
I20260925 10:37:29.646209 21839 sys_catalog.cc:458] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:29.647419 21828 sys_catalog.cc:564] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:29.652832 21842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:29.657330 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:29.641830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:43693 (local address 127.0.0.1:43235)
0925 10:37:29.642000 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:29.642007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:29.642025 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:37:29.642069 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:29.642073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:29.642140 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:29.642249 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:29.643784 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:29.644719 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:29.646576 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:29.647234 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:29.647365 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:29.650430 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:29.650435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:29.650441 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:29.650468 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:29.654421 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:29.655165 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:29.655171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:29.655174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:29.655316 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:29.655760 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:29.655765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:29.655768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:29.656099 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:29.656395 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:29.656797 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 10:37:29.656971 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:37:29.657793 21842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:29.666249 21842 catalog_manager.cc:1347] Generated new cluster ID: 4935dcdc365e449fafe4148e7cf3fcf9
I20260925 10:37:29.666386 21842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:29.704669 21842 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:29.706393 21842 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:29.714953 21842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f030e61a060441aea4d87eb336b799cc: Generated new TSK 0
I20260925 10:37:29.715756 21842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:29.729638 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 21761
I20260925 10:37:29.731302 21823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:29.825119 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 21761
W20260925 10:37:29.842025 21746 connection.cc:535] client connection to 127.0.0.1:43235 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43235): Connection reset by peer (error 104)
2026-09-25T10:37:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1963 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:29 dist-test-slave-dxdd krb5kdc[21860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:29 dist-test-slave-dxdd krb5kdc[21860](info): set up 2 sockets
Sep 25 10:37:29 dist-test-slave-dxdd krb5kdc[21860](info): commencing operation
krb5kdc: starting...
W20260925 10:37:30.910542 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:31 dist-test-slave-dxdd krb5kdc[21860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:37:31.155295 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:31.471024 21873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:31.471289 21873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:31.471377 21873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:31.478205 21873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:31.478317 21873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:31.478401 21873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:31.478466 21873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:31.478513 21873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:31.487493 21873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:31.488646 21873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:31.489863 21873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:31.501398 21878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:31.501369 21879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:31.501449 21881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:31.502157 21873 server_base.cc:1034] running on GCE node
I20260925 10:37:31.502954 21873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:31.504455 21873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:31.505657 21873 hybrid_clock.cc:648] HybridClock initialized: now 1790332651505614 us; error 37 us; skew 500 ppm
I20260925 10:37:31.506001 21873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:31 dist-test-slave-dxdd krb5kdc[21860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:31.509058 21873 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:37:31.510490 21873 webserver.cc:460] Webserver started at http://127.0.0.1:39055/ using document root <none> and password file <none>
I20260925 10:37:31.511128 21873 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:31.511224 21873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:31.511459 21873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:31.514205 21873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "331b566f3d3743909d7f6492d91e1b82"
format_stamp: "Formatted at 2026-09-25 10:37:31 on dist-test-slave-dxdd"
server_key: "1d10ceda0a3284a77f66818061dba727"
server_key_iv: "b1962fb682faf19ef869227a292c004e"
server_key_version: "encryptionkey@0"
I20260925 10:37:31.514943 21873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "331b566f3d3743909d7f6492d91e1b82"
format_stamp: "Formatted at 2026-09-25 10:37:31 on dist-test-slave-dxdd"
server_key: "1d10ceda0a3284a77f66818061dba727"
server_key_iv: "b1962fb682faf19ef869227a292c004e"
server_key_version: "encryptionkey@0"
I20260925 10:37:31.519665 21873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 10:37:31.522840 21888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:31.524147 21873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 10:37:31.524278 21873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "331b566f3d3743909d7f6492d91e1b82"
format_stamp: "Formatted at 2026-09-25 10:37:31 on dist-test-slave-dxdd"
server_key: "1d10ceda0a3284a77f66818061dba727"
server_key_iv: "b1962fb682faf19ef869227a292c004e"
server_key_version: "encryptionkey@0"
I20260925 10:37:31.524420 21873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:31.539258 21873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:31.543315 21873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:31.543530 21873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:31.561342 21873 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42201
I20260925 10:37:31.561384 21939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42201 every 8 connection(s)
I20260925 10:37:31.562625 21873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:31.566013 21940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:31.566535 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 21873
I20260925 10:37:31.566700 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:31.567116 388 external_mini_cluster.cc:1428] Setting key 373ae4f02018ae8d554cabaa4bf18d0d
I20260925 10:37:31.577535 21940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82: Bootstrap starting.
I20260925 10:37:31.579886 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:31.568694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:38809 (local address 127.0.0.1:42201)
0925 10:37:31.569402 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:31.569412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:31.569449 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:37:31.571746 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:31.571767 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:31.572144 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:31.572521 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:31.573019 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:31.574215 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:31.575563 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:31.576234 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:31.576426 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:31.577747 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:31.577761 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:31.577774 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:31.578339 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:31.578423 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:31.578450 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:31.578604 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:31.578866 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 10:37:31.579036 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":337,"threads_started":1}
I20260925 10:37:31.580986 21940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:31.582281 21940 log.cc:824] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:31.583776 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:31.584916 21940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82: No bootstrap required, opened a new log
I20260925 10:37:31.594532 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:31.588103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:42839 (local address 127.0.0.1:42201)
0925 10:37:31.588284 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:31.588290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:31.588659 (+ 369us) server_negotiation.cc:407] Connection header received
0925 10:37:31.588800 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:31.588805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:31.588880 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:31.588999 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:31.589504 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:31.590657 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:31.591947 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:31.592684 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:31.592793 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:31.593783 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:31.593790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:31.593795 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:31.593815 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:31.593879 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:31.593910 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:31.594026 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:31.594137 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 10:37:31.594192 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:37:31.595323 21940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331b566f3d3743909d7f6492d91e1b82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42201 } }
I20260925 10:37:31.595750 21940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:31.595868 21940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 331b566f3d3743909d7f6492d91e1b82, State: Initialized, Role: FOLLOWER
I20260925 10:37:31.596421 21940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331b566f3d3743909d7f6492d91e1b82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42201 } }
I20260925 10:37:31.596621 21940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:31.596710 21940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:31.596822 21940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:31.599359 21940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331b566f3d3743909d7f6492d91e1b82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42201 } }
I20260925 10:37:31.599882 21940 leader_election.cc:302] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 331b566f3d3743909d7f6492d91e1b82; no voters:
I20260925 10:37:31.600373 21940 leader_election.cc:288] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:31.600492 21950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:31.601245 21950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [term 1 LEADER]: Becoming Leader. State: Replica: 331b566f3d3743909d7f6492d91e1b82, State: Running, Role: LEADER
I20260925 10:37:31.601801 21950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331b566f3d3743909d7f6492d91e1b82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42201 } }
I20260925 10:37:31.602519 21940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:31.604396 21952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 331b566f3d3743909d7f6492d91e1b82. Latest consensus state: current_term: 1 leader_uuid: "331b566f3d3743909d7f6492d91e1b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331b566f3d3743909d7f6492d91e1b82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42201 } } }
I20260925 10:37:31.604728 21952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:31.605197 21951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "331b566f3d3743909d7f6492d91e1b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331b566f3d3743909d7f6492d91e1b82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42201 } } }
I20260925 10:37:31.605562 21951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:31.610793 21955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:31.617723 21955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:31.628589 21955 catalog_manager.cc:1347] Generated new cluster ID: e3ebdef134e4478db16c3d6ac3e26980
I20260925 10:37:31.628693 21955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:31.661525 21955 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:31.663408 21955 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:31.670194 21955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 331b566f3d3743909d7f6492d91e1b82: Generated new TSK 0
I20260925 10:37:31.671108 21955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:31.689487 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 21873
W20260925 10:37:31.690536 21935 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.183:38809
W20260925 10:37:31.691205 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 21873: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:37:31.691349 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 21873
W20260925 10:37:31.707917 21858 connection.cc:535] client connection to 127.0.0.1:42201 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42201): Connection reset by peer (error 104)
2026-09-25T10:37:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:31 dist-test-slave-dxdd krb5kdc[21972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:31 dist-test-slave-dxdd krb5kdc[21972](info): set up 2 sockets
Sep 25 10:37:31 dist-test-slave-dxdd krb5kdc[21972](info): commencing operation
krb5kdc: starting...
W20260925 10:37:32.778648 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:32 dist-test-slave-dxdd krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:37:33.004920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:33.318511 21985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:33.318830 21985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:33.318993 21985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:33.326007 21985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:33.326157 21985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:33.326269 21985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:33.326390 21985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:33.326463 21985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:33.335726 21985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:33.336946 21985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:33.338234 21985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:33.349195 21991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:33.349268 21990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:33.349244 21993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:33.350126 21985 server_base.cc:1034] running on GCE node
I20260925 10:37:33.350875 21985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:33.352130 21985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:33.353318 21985 hybrid_clock.cc:648] HybridClock initialized: now 1790332653353278 us; error 28 us; skew 500 ppm
I20260925 10:37:33.353657 21985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:33 dist-test-slave-dxdd krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:33.363018 21985 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:37:33.364456 21985 webserver.cc:460] Webserver started at http://127.0.0.1:38703/ using document root <none> and password file <none>
I20260925 10:37:33.365106 21985 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:33.365245 21985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:33.365541 21985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:33.368427 21985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "f22a348407234cb59442786f9e64b745"
format_stamp: "Formatted at 2026-09-25 10:37:33 on dist-test-slave-dxdd"
server_key: "9283c4f68b694d7999c303766f2986fd"
server_key_iv: "f7da54f696bcbf33922af7c0adcbd94b"
server_key_version: "encryptionkey@0"
I20260925 10:37:33.369155 21985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "f22a348407234cb59442786f9e64b745"
format_stamp: "Formatted at 2026-09-25 10:37:33 on dist-test-slave-dxdd"
server_key: "9283c4f68b694d7999c303766f2986fd"
server_key_iv: "f7da54f696bcbf33922af7c0adcbd94b"
server_key_version: "encryptionkey@0"
I20260925 10:37:33.373844 21985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260925 10:37:33.377017 22000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:33.378203 21985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:37:33.378394 21985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "f22a348407234cb59442786f9e64b745"
format_stamp: "Formatted at 2026-09-25 10:37:33 on dist-test-slave-dxdd"
server_key: "9283c4f68b694d7999c303766f2986fd"
server_key_iv: "f7da54f696bcbf33922af7c0adcbd94b"
server_key_version: "encryptionkey@0"
I20260925 10:37:33.378582 21985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:33.383919 21985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:33.387521 21985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:33.387789 21985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:33.404960 21985 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39305
I20260925 10:37:33.405028 22051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39305 every 8 connection(s)
I20260925 10:37:33.406224 21985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:33.409605 22052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:33.415524 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 21985
I20260925 10:37:33.415688 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:33.416103 388 external_mini_cluster.cc:1428] Setting key b8a9eedca1436753b3e9295c4503acd7
I20260925 10:37:33.419654 22052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745: Bootstrap starting.
I20260925 10:37:33.422773 22052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:33.423301 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:33.417748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:43311 (local address 127.0.0.1:39305)
0925 10:37:33.418504 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:33.418514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:33.418550 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:37:33.420176 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:33.420196 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:33.420490 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:33.420844 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:33.421779 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:33.421785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:33.421797 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:33.421809 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:33.421865 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:33.421882 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:33.422087 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:33.422222 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 10:37:33.422437 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":305,"threads_started":1}
I20260925 10:37:33.423918 22052 log.cc:824] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:33.426800 22052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745: No bootstrap required, opened a new log
I20260925 10:37:33.426841 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:33.433251 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:33.431072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:50417 (local address 127.0.0.1:39305)
0925 10:37:33.431247 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:33.431252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:33.431492 (+ 240us) server_negotiation.cc:407] Connection header received
0925 10:37:33.431662 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:33.431666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:33.431715 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:33.431813 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:33.432648 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:33.432653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:33.432658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:33.432677 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:33.432722 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:33.432736 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:33.432862 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:33.432948 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 10:37:33.433005 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 10:37:33.434458 22052 raft_consensus.cc:348] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22a348407234cb59442786f9e64b745" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } }
I20260925 10:37:33.434844 22052 raft_consensus.cc:374] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:33.434942 22052 raft_consensus.cc:729] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f22a348407234cb59442786f9e64b745, State: Initialized, Role: FOLLOWER
I20260925 10:37:33.435443 22052 consensus_queue.cc:260] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22a348407234cb59442786f9e64b745" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } }
I20260925 10:37:33.435678 22052 raft_consensus.cc:388] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:33.435813 22052 raft_consensus.cc:482] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:33.435981 22052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:33.438021 22052 raft_consensus.cc:504] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22a348407234cb59442786f9e64b745" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } }
I20260925 10:37:33.438566 22052 leader_election.cc:302] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f22a348407234cb59442786f9e64b745; no voters:
I20260925 10:37:33.439097 22052 leader_election.cc:288] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:33.439203 22062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:33.439838 22062 raft_consensus.cc:686] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [term 1 LEADER]: Becoming Leader. State: Replica: f22a348407234cb59442786f9e64b745, State: Running, Role: LEADER
I20260925 10:37:33.440402 22062 consensus_queue.cc:237] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22a348407234cb59442786f9e64b745" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } }
I20260925 10:37:33.441148 22052 sys_catalog.cc:564] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:33.443146 22064 sys_catalog.cc:455] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f22a348407234cb59442786f9e64b745. Latest consensus state: current_term: 1 leader_uuid: "f22a348407234cb59442786f9e64b745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22a348407234cb59442786f9e64b745" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } } }
I20260925 10:37:33.443151 22063 sys_catalog.cc:455] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f22a348407234cb59442786f9e64b745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22a348407234cb59442786f9e64b745" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } } }
I20260925 10:37:33.443711 22063 sys_catalog.cc:458] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:33.443739 22064 sys_catalog.cc:458] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:33.445319 22068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:33.449976 22068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:33.460405 22068 catalog_manager.cc:1347] Generated new cluster ID: a8be4223783a4cec9e4dd1d5f99facc5
I20260925 10:37:33.460518 22068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:33.485057 22068 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:33.485493 22068 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:33.495657 22068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f22a348407234cb59442786f9e64b745: Generated new TSK 0
I20260925 10:37:33.496332 22068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:33.509770 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 21985
W20260925 10:37:33.511055 22047 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.183:43311
W20260925 10:37:33.511706 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 21985: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:37:33.511847 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 21985
2026-09-25T10:37:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1816 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:33 dist-test-slave-dxdd krb5kdc[22083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:33 dist-test-slave-dxdd krb5kdc[22083](info): set up 2 sockets
Sep 25 10:37:33 dist-test-slave-dxdd krb5kdc[22083](info): commencing operation
krb5kdc: starting...
W20260925 10:37:34.598789 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:34 dist-test-slave-dxdd krb5kdc[22083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:37:34.841046 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44609
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:35.164567 22096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:35.164840 22096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:35.164963 22096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:35.171916 22096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:35.172024 22096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:35.172135 22096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:35.172238 22096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:35.172312 22096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:35.181937 22096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44609
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:35.183233 22096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:35.184458 22096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:35.195983 22101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:35.196014 22102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:35.195937 22104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:35.196758 22096 server_base.cc:1034] running on GCE node
I20260925 10:37:35.197468 22096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:35.198904 22096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:35.200112 22096 hybrid_clock.cc:648] HybridClock initialized: now 1790332655200064 us; error 50 us; skew 500 ppm
I20260925 10:37:35.200426 22096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:35 dist-test-slave-dxdd krb5kdc[22083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:35.203510 22096 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:37:35.204813 22096 webserver.cc:460] Webserver started at http://127.0.97.62:45699/ using document root <none> and password file <none>
I20260925 10:37:35.205407 22096 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:35.205500 22096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:35.205747 22096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:35.208561 22096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "9b86246200c54bb693db2ab3a660f94b"
format_stamp: "Formatted at 2026-09-25 10:37:35 on dist-test-slave-dxdd"
server_key: "f407d126b287c653ede2ebb7aed986e6"
server_key_iv: "0c3d4959562a08f667b666d9ba629aba"
server_key_version: "encryptionkey@0"
I20260925 10:37:35.209290 22096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "9b86246200c54bb693db2ab3a660f94b"
format_stamp: "Formatted at 2026-09-25 10:37:35 on dist-test-slave-dxdd"
server_key: "f407d126b287c653ede2ebb7aed986e6"
server_key_iv: "0c3d4959562a08f667b666d9ba629aba"
server_key_version: "encryptionkey@0"
I20260925 10:37:35.214118 22096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260925 10:37:35.217285 22111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:35.218536 22096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:37:35.218662 22096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "9b86246200c54bb693db2ab3a660f94b"
format_stamp: "Formatted at 2026-09-25 10:37:35 on dist-test-slave-dxdd"
server_key: "f407d126b287c653ede2ebb7aed986e6"
server_key_iv: "0c3d4959562a08f667b666d9ba629aba"
server_key_version: "encryptionkey@0"
I20260925 10:37:35.218811 22096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:35.229566 22096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:35.233069 22096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:35.233320 22096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:35.250839 22096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44609
I20260925 10:37:35.250876 22162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44609 every 8 connection(s)
I20260925 10:37:35.252089 22096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:35.253906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 22096
I20260925 10:37:35.254081 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:35.254565 388 external_mini_cluster.cc:1428] Setting key de2dfb0c98adec79c7c8c19d84f3accc
I20260925 10:37:35.255733 22163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:35.266490 22163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b: Bootstrap starting.
Sep 25 10:37:35 dist-test-slave-dxdd krb5kdc[22083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:37:35.269454 22163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:35.270593 22163 log.cc:824] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:35.273138 22163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b: No bootstrap required, opened a new log
I20260925 10:37:35.275030 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:35.256176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47943 (local address 127.0.97.62:44609)
0925 10:37:35.256890 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:35.256901 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:35.256938 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:37:35.258513 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:35.258533 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:35.258892 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:35.259287 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:35.260587 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:35.261893 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:35.263208 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:35.263901 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:35.264100 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:35.268137 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:35.268145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:35.268158 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:35.268201 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:35.271212 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:35.271929 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:35.271936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:35.271941 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:35.272098 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:35.272540 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:35.272550 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:35.272552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:35.272977 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:35.273306 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:35.273746 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 10:37:35.274044 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":313,"threads_started":1}
I20260925 10:37:35.278506 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:35.280879 22163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b86246200c54bb693db2ab3a660f94b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44609 } }
I20260925 10:37:35.281286 22163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:35.281416 22163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b86246200c54bb693db2ab3a660f94b, State: Initialized, Role: FOLLOWER
I20260925 10:37:35.281993 22163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b86246200c54bb693db2ab3a660f94b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44609 } }
I20260925 10:37:35.282271 22163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:35.282480 22163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:35.282678 22163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:35.284855 22163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b86246200c54bb693db2ab3a660f94b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44609 } }
I20260925 10:37:35.285326 22163 leader_election.cc:302] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b86246200c54bb693db2ab3a660f94b; no voters:
I20260925 10:37:35.285938 22163 leader_election.cc:288] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:35.286032 22173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:35.287827 22173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [term 1 LEADER]: Becoming Leader. State: Replica: 9b86246200c54bb693db2ab3a660f94b, State: Running, Role: LEADER
I20260925 10:37:35.288429 22173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b86246200c54bb693db2ab3a660f94b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44609 } }
I20260925 10:37:35.288861 22163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:35.294060 22175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b86246200c54bb693db2ab3a660f94b. Latest consensus state: current_term: 1 leader_uuid: "9b86246200c54bb693db2ab3a660f94b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b86246200c54bb693db2ab3a660f94b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44609 } } }
I20260925 10:37:35.294445 22175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:35.295097 22174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b86246200c54bb693db2ab3a660f94b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b86246200c54bb693db2ab3a660f94b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44609 } } }
I20260925 10:37:35.295428 22174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:35.299777 22183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:35.300961 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:35.282547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58205 (local address 127.0.97.62:44609)
0925 10:37:35.282725 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:35.282729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:35.283182 (+ 453us) server_negotiation.cc:407] Connection header received
0925 10:37:35.283369 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:35.283374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:35.283449 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:35.283566 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:35.284852 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:35.285731 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:35.287620 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:35.288225 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:35.288357 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:35.291095 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:35.291107 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:35.291114 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:35.291160 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:35.295907 (+ 4747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:35.297137 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:35.297145 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:35.297151 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:35.297403 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:35.298092 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:35.298099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:35.298105 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:35.298756 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:35.300236 (+ 1480us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:35.300399 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 10:37:35.300582 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 10:37:35.305074 22183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:35.314031 22183 catalog_manager.cc:1347] Generated new cluster ID: 46f7dc9cee1f4e1bbc8ab6048b98de0b
I20260925 10:37:35.314129 22183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:35.329463 22183 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:35.330699 22183 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:35.341252 22183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b86246200c54bb693db2ab3a660f94b: Generated new TSK 0
I20260925 10:37:35.342087 22183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:35.353821 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 22096
I20260925 10:37:35.355065 22158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:35.445206 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 22096
W20260925 10:37:35.461812 22081 connection.cc:535] client connection to 127.0.97.62:44609 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:44609): Connection reset by peer (error 104)
2026-09-25T10:37:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1932 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:35 dist-test-slave-dxdd krb5kdc[22196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:35 dist-test-slave-dxdd krb5kdc[22196](info): set up 2 sockets
Sep 25 10:37:35 dist-test-slave-dxdd krb5kdc[22196](info): commencing operation
krb5kdc: starting...
W20260925 10:37:36.530725 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:36 dist-test-slave-dxdd krb5kdc[22196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:37:36.762676 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41813
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:37.087450 22209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:37.087798 22209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:37.087967 22209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:37.094879 22209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:37.095014 22209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:37.095117 22209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:37.095220 22209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:37.095305 22209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:37.104538 22209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41813
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:37.105734 22209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:37.106969 22209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:37.118170 22214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:37.118134 22217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:37.118095 22215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:37.119309 22209 server_base.cc:1034] running on GCE node
I20260925 10:37:37.119999 22209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:37.121270 22209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:37.122483 22209 hybrid_clock.cc:648] HybridClock initialized: now 1790332657122436 us; error 43 us; skew 500 ppm
I20260925 10:37:37.122814 22209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:37 dist-test-slave-dxdd krb5kdc[22196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:37.125749 22209 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:37:37.127086 22209 webserver.cc:460] Webserver started at http://127.0.97.62:37903/ using document root <none> and password file <none>
I20260925 10:37:37.127758 22209 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:37.127887 22209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:37.128175 22209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:37.131001 22209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "faffb0bd4958454eb107b3d12a19c9ec"
format_stamp: "Formatted at 2026-09-25 10:37:37 on dist-test-slave-dxdd"
server_key: "fbd8e33e083f67b3a87f8c1188f9026d"
server_key_iv: "0532810ae7483a04c919e5df0671f332"
server_key_version: "encryptionkey@0"
I20260925 10:37:37.131732 22209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "faffb0bd4958454eb107b3d12a19c9ec"
format_stamp: "Formatted at 2026-09-25 10:37:37 on dist-test-slave-dxdd"
server_key: "fbd8e33e083f67b3a87f8c1188f9026d"
server_key_iv: "0532810ae7483a04c919e5df0671f332"
server_key_version: "encryptionkey@0"
I20260925 10:37:37.136466 22209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:37:37.139792 22224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:37.141199 22209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:37:37.141419 22209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "faffb0bd4958454eb107b3d12a19c9ec"
format_stamp: "Formatted at 2026-09-25 10:37:37 on dist-test-slave-dxdd"
server_key: "fbd8e33e083f67b3a87f8c1188f9026d"
server_key_iv: "0532810ae7483a04c919e5df0671f332"
server_key_version: "encryptionkey@0"
I20260925 10:37:37.141608 22209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:37.163303 22209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:37.167093 22209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:37.167376 22209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:37.185149 22209 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41813
I20260925 10:37:37.185184 22275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41813 every 8 connection(s)
I20260925 10:37:37.186544 22209 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:37.190025 22276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:37.195830 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 22209
I20260925 10:37:37.196005 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:37.196414 388 external_mini_cluster.cc:1428] Setting key d1f2c91422154d998255a63ba2d32847
I20260925 10:37:37.200883 22276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec: Bootstrap starting.
I20260925 10:37:37.204723 22276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:37.205787 22276 log.cc:824] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:37.208881 22276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec: No bootstrap required, opened a new log
I20260925 10:37:37.211050 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:37.198207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53517 (local address 127.0.97.62:41813)
0925 10:37:37.199062 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:37.199077 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:37.199123 (+ 46us) server_negotiation.cc:407] Connection header received
0925 10:37:37.201390 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:37.201420 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:37.201885 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:37.202296 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:37.202850 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:37.204784 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:37.206197 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:37.207080 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:37.207252 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:37.208441 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:37.208452 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:37.208469 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:37.209144 (+ 675us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:37.209230 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:37.209262 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:37.209444 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:37.209652 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 10:37:37.209815 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":348,"threads_started":1}
I20260925 10:37:37.214826 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:37.217850 22276 raft_consensus.cc:348] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faffb0bd4958454eb107b3d12a19c9ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41813 } }
I20260925 10:37:37.218214 22276 raft_consensus.cc:374] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:37.218325 22276 raft_consensus.cc:729] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faffb0bd4958454eb107b3d12a19c9ec, State: Initialized, Role: FOLLOWER
I20260925 10:37:37.218978 22276 consensus_queue.cc:260] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faffb0bd4958454eb107b3d12a19c9ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41813 } }
I20260925 10:37:37.219287 22276 raft_consensus.cc:388] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:37.219431 22276 raft_consensus.cc:482] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:37.219614 22276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:37.222564 22276 raft_consensus.cc:504] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faffb0bd4958454eb107b3d12a19c9ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41813 } }
I20260925 10:37:37.223150 22276 leader_election.cc:302] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faffb0bd4958454eb107b3d12a19c9ec; no voters:
I20260925 10:37:37.223821 22276 leader_election.cc:288] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:37.223937 22286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:37.225379 22286 raft_consensus.cc:686] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [term 1 LEADER]: Becoming Leader. State: Replica: faffb0bd4958454eb107b3d12a19c9ec, State: Running, Role: LEADER
I20260925 10:37:37.225872 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:37.218874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59039 (local address 127.0.97.62:41813)
0925 10:37:37.219060 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:37.219067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:37.219586 (+ 519us) server_negotiation.cc:407] Connection header received
0925 10:37:37.219821 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:37.219825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:37.219879 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:37.219956 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:37.220509 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:37.221258 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:37.223050 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:37.223532 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:37.223663 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:37.224920 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:37.224927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:37.224931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:37.224950 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:37.225015 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:37.225033 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:37.225135 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:37.225531 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 10:37:37.225579 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:37:37.226172 22276 sys_catalog.cc:564] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:37.226217 22286 consensus_queue.cc:237] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faffb0bd4958454eb107b3d12a19c9ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41813 } }
I20260925 10:37:37.228905 22288 sys_catalog.cc:455] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader faffb0bd4958454eb107b3d12a19c9ec. Latest consensus state: current_term: 1 leader_uuid: "faffb0bd4958454eb107b3d12a19c9ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faffb0bd4958454eb107b3d12a19c9ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41813 } } }
I20260925 10:37:37.229228 22288 sys_catalog.cc:458] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:37.229022 22287 sys_catalog.cc:455] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faffb0bd4958454eb107b3d12a19c9ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faffb0bd4958454eb107b3d12a19c9ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41813 } } }
I20260925 10:37:37.229408 22287 sys_catalog.cc:458] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:37.230055 22294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:37.235214 22294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:37.245303 22294 catalog_manager.cc:1347] Generated new cluster ID: ea8e6bb798e54648b3a19bafd981a724
I20260925 10:37:37.245404 22294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:37.266080 22294 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:37.267988 22294 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:37.276844 22294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faffb0bd4958454eb107b3d12a19c9ec: Generated new TSK 0
I20260925 10:37:37.277654 22294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:37.282593 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 22209
W20260925 10:37:37.283882 22271 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53517
W20260925 10:37:37.284574 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 22209: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:37:37.284710 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 22209
W20260925 10:37:37.300943 22194 connection.cc:535] client connection to 127.0.97.62:41813 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:41813): Connection reset by peer (error 104)
2026-09-25T10:37:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1839 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:37 dist-test-slave-dxdd krb5kdc[22307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:37 dist-test-slave-dxdd krb5kdc[22307](info): set up 2 sockets
Sep 25 10:37:37 dist-test-slave-dxdd krb5kdc[22307](info): commencing operation
krb5kdc: starting...
W20260925 10:37:38.362735 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:38 dist-test-slave-dxdd krb5kdc[22307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:37:38.593755 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45259
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:38.910333 22320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:38.910660 22320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:38.910828 22320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:38.918020 22320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:38.918180 22320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:38.918283 22320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:38.918402 22320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:38.918480 22320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:38.928061 22320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:38.929276 22320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:38.930550 22320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:38.942081 22325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:38.942124 22328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:38.942093 22326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:38.942906 22320 server_base.cc:1034] running on GCE node
I20260925 10:37:38.943614 22320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:38.944868 22320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:38.946089 22320 hybrid_clock.cc:648] HybridClock initialized: now 1790332658946054 us; error 45 us; skew 500 ppm
I20260925 10:37:38.946480 22320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:38 dist-test-slave-dxdd krb5kdc[22307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:38.953056 22320 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:37:38.954608 22320 webserver.cc:460] Webserver started at http://127.0.97.62:43547/ using document root <none> and password file <none>
I20260925 10:37:38.955262 22320 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:38.955400 22320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:38.955691 22320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:38.958660 22320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "d85645face52499f9ce644d061cc6f0d"
format_stamp: "Formatted at 2026-09-25 10:37:38 on dist-test-slave-dxdd"
server_key: "76b789a7cb884a3f6157a267c3e7e0b6"
server_key_iv: "e42ec49ee11b0b98982162e14c155d00"
server_key_version: "encryptionkey@0"
I20260925 10:37:38.959422 22320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "d85645face52499f9ce644d061cc6f0d"
format_stamp: "Formatted at 2026-09-25 10:37:38 on dist-test-slave-dxdd"
server_key: "76b789a7cb884a3f6157a267c3e7e0b6"
server_key_iv: "e42ec49ee11b0b98982162e14c155d00"
server_key_version: "encryptionkey@0"
I20260925 10:37:38.964298 22320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260925 10:37:38.967634 22335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:38.968822 22320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:37:38.968990 22320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "d85645face52499f9ce644d061cc6f0d"
format_stamp: "Formatted at 2026-09-25 10:37:38 on dist-test-slave-dxdd"
server_key: "76b789a7cb884a3f6157a267c3e7e0b6"
server_key_iv: "e42ec49ee11b0b98982162e14c155d00"
server_key_version: "encryptionkey@0"
I20260925 10:37:38.969185 22320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:38.974707 22320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:38.978346 22320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:38.978641 22320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:38.996004 22320 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45259
I20260925 10:37:38.996021 22386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45259 every 8 connection(s)
I20260925 10:37:38.997270 22320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:39.000756 22387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:39.006434 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 22320
I20260925 10:37:39.006590 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:39.007025 388 external_mini_cluster.cc:1428] Setting key 5c9da38de1a260154b7d884de9cdca9c
I20260925 10:37:39.011086 22387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d: Bootstrap starting.
I20260925 10:37:39.014551 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:39.008717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33445 (local address 127.0.97.62:45259)
0925 10:37:39.009546 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:39.009556 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:39.009595 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:37:39.011274 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:39.011307 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:39.011621 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:39.011945 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:39.013031 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:39.013037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:39.013049 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:39.013061 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:39.013122 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:39.013139 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:39.013295 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:39.013403 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 10:37:39.013582 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":404,"threads_started":1}
I20260925 10:37:39.014649 22387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:39.015981 22387 log.cc:824] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:39.018399 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:39.018983 22387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d: No bootstrap required, opened a new log
I20260925 10:37:39.025290 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:39.022911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53871 (local address 127.0.97.62:45259)
0925 10:37:39.023103 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:39.023110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:39.023443 (+ 333us) server_negotiation.cc:407] Connection header received
0925 10:37:39.023628 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:39.023633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:39.023691 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:39.023800 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:39.024673 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:39.024679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:39.024685 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:39.024702 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:39.024747 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:39.024762 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:39.024900 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:39.024963 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 10:37:39.025027 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:37:39.026926 22387 raft_consensus.cc:348] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85645face52499f9ce644d061cc6f0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45259 } }
I20260925 10:37:39.027254 22387 raft_consensus.cc:374] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:39.027359 22387 raft_consensus.cc:729] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85645face52499f9ce644d061cc6f0d, State: Initialized, Role: FOLLOWER
I20260925 10:37:39.027987 22387 consensus_queue.cc:260] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85645face52499f9ce644d061cc6f0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45259 } }
I20260925 10:37:39.028276 22387 raft_consensus.cc:388] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:39.028424 22387 raft_consensus.cc:482] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:39.028611 22387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:39.030792 22387 raft_consensus.cc:504] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85645face52499f9ce644d061cc6f0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45259 } }
I20260925 10:37:39.031226 22387 leader_election.cc:302] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d85645face52499f9ce644d061cc6f0d; no voters:
I20260925 10:37:39.031708 22387 leader_election.cc:288] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:39.031827 22397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:39.033138 22397 raft_consensus.cc:686] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [term 1 LEADER]: Becoming Leader. State: Replica: d85645face52499f9ce644d061cc6f0d, State: Running, Role: LEADER
I20260925 10:37:39.033594 22397 consensus_queue.cc:237] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85645face52499f9ce644d061cc6f0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45259 } }
I20260925 10:37:39.033707 22387 sys_catalog.cc:564] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:39.035925 22399 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d85645face52499f9ce644d061cc6f0d. Latest consensus state: current_term: 1 leader_uuid: "d85645face52499f9ce644d061cc6f0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85645face52499f9ce644d061cc6f0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45259 } } }
I20260925 10:37:39.036239 22399 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:39.036800 22398 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d85645face52499f9ce644d061cc6f0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85645face52499f9ce644d061cc6f0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45259 } } }
I20260925 10:37:39.037091 22398 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:39.037091 22404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:39.049283 22404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:39.060060 22404 catalog_manager.cc:1347] Generated new cluster ID: 62a3a8f383024729a9149e572193b16b
I20260925 10:37:39.060163 22404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:39.076651 22404 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:39.077041 22404 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:39.085059 22404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d85645face52499f9ce644d061cc6f0d: Generated new TSK 0
I20260925 10:37:39.085804 22404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:39.094794 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 22320
W20260925 10:37:39.095988 22382 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33445
W20260925 10:37:39.096658 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 22320: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:37:39.096788 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 22320
2026-09-25T10:37:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1812 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:39 dist-test-slave-dxdd krb5kdc[22419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:39 dist-test-slave-dxdd krb5kdc[22419](info): set up 2 sockets
Sep 25 10:37:39 dist-test-slave-dxdd krb5kdc[22419](info): commencing operation
krb5kdc: starting...
W20260925 10:37:40.178690 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:40 dist-test-slave-dxdd krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:37:40.422734 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33379
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:40.736704 22432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:40.737090 22432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:40.737250 22432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:40.744374 22432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:40.744506 22432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:40.744596 22432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:40.744714 22432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:40.744792 22432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:40.754146 22432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:40.755389 22432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:40.756628 22432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:40.767957 22438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:40.768007 22440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:40.768121 22437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:40.768664 22432 server_base.cc:1034] running on GCE node
I20260925 10:37:40.769402 22432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:40.770699 22432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:40.771914 22432 hybrid_clock.cc:648] HybridClock initialized: now 1790332660771878 us; error 45 us; skew 500 ppm
I20260925 10:37:40.772248 22432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:40 dist-test-slave-dxdd krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:40.775221 22432 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:37:40.776508 22432 webserver.cc:460] Webserver started at http://127.0.97.62:45711/ using document root <none> and password file <none>
I20260925 10:37:40.777107 22432 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:40.777227 22432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:40.777462 22432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:40.780228 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "49c29a9f18b447b480b5585ec92f9061"
format_stamp: "Formatted at 2026-09-25 10:37:40 on dist-test-slave-dxdd"
server_key: "5d37e428468c7a79cfc065bd0fbdb6f2"
server_key_iv: "ce0355253c137ecf2f515758fe16ada7"
server_key_version: "encryptionkey@0"
I20260925 10:37:40.780893 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "49c29a9f18b447b480b5585ec92f9061"
format_stamp: "Formatted at 2026-09-25 10:37:40 on dist-test-slave-dxdd"
server_key: "5d37e428468c7a79cfc065bd0fbdb6f2"
server_key_iv: "ce0355253c137ecf2f515758fe16ada7"
server_key_version: "encryptionkey@0"
I20260925 10:37:40.785609 22432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 10:37:40.788710 22447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:40.789847 22432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:37:40.790007 22432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "49c29a9f18b447b480b5585ec92f9061"
format_stamp: "Formatted at 2026-09-25 10:37:40 on dist-test-slave-dxdd"
server_key: "5d37e428468c7a79cfc065bd0fbdb6f2"
server_key_iv: "ce0355253c137ecf2f515758fe16ada7"
server_key_version: "encryptionkey@0"
I20260925 10:37:40.790187 22432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:40.819576 22432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:40.823993 22432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:40.824281 22432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:40.841192 22432 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33379
I20260925 10:37:40.841238 22498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33379 every 8 connection(s)
I20260925 10:37:40.842517 22432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:40.845901 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 22432
I20260925 10:37:40.846089 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:40.846230 22499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:40.846601 388 external_mini_cluster.cc:1428] Setting key 771dce026ca65053e5ea4f9725979cd8
I20260925 10:37:40.859905 22499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061: Bootstrap starting.
Sep 25 10:37:40 dist-test-slave-dxdd krb5kdc[22419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:37:40.862977 22499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:40.863981 22499 log.cc:824] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:40.866652 22499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061: No bootstrap required, opened a new log
I20260925 10:37:40.868680 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:40.848255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39345 (local address 127.0.97.62:33379)
0925 10:37:40.849048 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:40.849059 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:40.849096 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:37:40.850665 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:40.850687 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:40.851089 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:40.851397 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:40.853253 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:40.854735 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:40.856586 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:40.857284 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:40.857429 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:40.861913 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:40.861920 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:40.861932 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:40.861963 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:40.865043 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:40.865667 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:40.865673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:40.865679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:40.865834 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:40.866203 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:40.866212 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:40.866214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:40.866722 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:40.866975 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:40.867409 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 10:37:40.867734 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":380,"threads_started":1}
I20260925 10:37:40.872225 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:40.874995 22499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c29a9f18b447b480b5585ec92f9061" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33379 } }
I20260925 10:37:40.875326 22499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:40.875435 22499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c29a9f18b447b480b5585ec92f9061, State: Initialized, Role: FOLLOWER
I20260925 10:37:40.876001 22499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c29a9f18b447b480b5585ec92f9061" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33379 } }
I20260925 10:37:40.876367 22499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:40.876579 22499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:40.876770 22499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:40.878925 22499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c29a9f18b447b480b5585ec92f9061" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33379 } }
I20260925 10:37:40.879380 22499 leader_election.cc:302] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49c29a9f18b447b480b5585ec92f9061; no voters:
I20260925 10:37:40.879985 22499 leader_election.cc:288] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:40.880077 22509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:40.882380 22499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:40.887050 22509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [term 1 LEADER]: Becoming Leader. State: Replica: 49c29a9f18b447b480b5585ec92f9061, State: Running, Role: LEADER
I20260925 10:37:40.887725 22509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c29a9f18b447b480b5585ec92f9061" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33379 } }
I20260925 10:37:40.889954 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:40.876391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38207 (local address 127.0.97.62:33379)
0925 10:37:40.876528 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:40.876532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:40.877070 (+ 538us) server_negotiation.cc:407] Connection header received
0925 10:37:40.877260 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:40.877265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:40.877335 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:40.877437 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:40.878882 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:40.879674 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:40.881436 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:40.881867 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:40.882015 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:40.884387 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:40.884395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:40.884406 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:40.884444 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:40.887420 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:40.888006 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:40.888011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:40.888014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:40.888128 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:40.888475 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:40.888482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:40.888485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:40.888869 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:40.889130 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:40.889438 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 10:37:40.889638 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:37:40.897630 22510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49c29a9f18b447b480b5585ec92f9061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c29a9f18b447b480b5585ec92f9061" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33379 } } }
I20260925 10:37:40.897574 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49c29a9f18b447b480b5585ec92f9061. Latest consensus state: current_term: 1 leader_uuid: "49c29a9f18b447b480b5585ec92f9061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c29a9f18b447b480b5585ec92f9061" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33379 } } }
I20260925 10:37:40.898249 22510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:40.898245 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061 [sys.catalog]: This master's current role is: LEADER
W20260925 10:37:40.901896 22523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:37:40.902029 22523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:37:40.903730 22524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:40.905205 22524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:40.912061 22524 catalog_manager.cc:1347] Generated new cluster ID: f4c0bfeb27d34370862997c226bcd65c
I20260925 10:37:40.912158 22524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:40.924831 22524 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:40.926542 22524 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:40.940339 22524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49c29a9f18b447b480b5585ec92f9061: Generated new TSK 0
I20260925 10:37:40.941293 22524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:40.955595 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 22432
I20260925 10:37:40.956637 22494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:41.047926 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 22432
2026-09-25T10:37:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1954 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:41 dist-test-slave-dxdd krb5kdc[22531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:41 dist-test-slave-dxdd krb5kdc[22531](info): set up 2 sockets
Sep 25 10:37:41 dist-test-slave-dxdd krb5kdc[22531](info): commencing operation
krb5kdc: starting...
W20260925 10:37:42.146782 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:42 dist-test-slave-dxdd krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:37:42.389499 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36917
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:42.706730 22544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:42.706995 22544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:42.707193 22544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:42.714030 22544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:42.714123 22544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:42.714167 22544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:42.714217 22544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:42.714279 22544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:42.724295 22544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:42.725518 22544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:42.726919 22544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:42.738504 22549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:42.738504 22550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:42.738659 22552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:42.739303 22544 server_base.cc:1034] running on GCE node
I20260925 10:37:42.740042 22544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:42.741410 22544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:42.742621 22544 hybrid_clock.cc:648] HybridClock initialized: now 1790332662742591 us; error 41 us; skew 500 ppm
I20260925 10:37:42.742938 22544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:42 dist-test-slave-dxdd krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:42.745846 22544 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:37:42.747253 22544 webserver.cc:460] Webserver started at http://127.0.97.62:35869/ using document root <none> and password file <none>
I20260925 10:37:42.747884 22544 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:42.748024 22544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:42.748329 22544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:42.751221 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "40f1d13cda9c44c89724b74a0f9cd3ff"
format_stamp: "Formatted at 2026-09-25 10:37:42 on dist-test-slave-dxdd"
server_key: "cf3782f81d6836099a4e85d1534fbec6"
server_key_iv: "ee4fa100314844de437e0fc3c91257dd"
server_key_version: "encryptionkey@0"
I20260925 10:37:42.751943 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "40f1d13cda9c44c89724b74a0f9cd3ff"
format_stamp: "Formatted at 2026-09-25 10:37:42 on dist-test-slave-dxdd"
server_key: "cf3782f81d6836099a4e85d1534fbec6"
server_key_iv: "ee4fa100314844de437e0fc3c91257dd"
server_key_version: "encryptionkey@0"
I20260925 10:37:42.756883 22544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:37:42.760206 22559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:42.761466 22544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:37:42.761633 22544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "40f1d13cda9c44c89724b74a0f9cd3ff"
format_stamp: "Formatted at 2026-09-25 10:37:42 on dist-test-slave-dxdd"
server_key: "cf3782f81d6836099a4e85d1534fbec6"
server_key_iv: "ee4fa100314844de437e0fc3c91257dd"
server_key_version: "encryptionkey@0"
I20260925 10:37:42.761819 22544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:42.782281 22544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:42.786099 22544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:42.786341 22544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:42.804236 22544 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36917
I20260925 10:37:42.804275 22610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36917 every 8 connection(s)
I20260925 10:37:42.805502 22544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:42.808990 22611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:42.812377 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 22544
I20260925 10:37:42.812561 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:42.812983 388 external_mini_cluster.cc:1428] Setting key e51da8d237421c23b064affb796594ec
I20260925 10:37:42.821714 22611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff: Bootstrap starting.
I20260925 10:37:42.825526 22611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:42.826812 22611 log.cc:824] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:42.827214 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:42.814671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35387 (local address 127.0.97.62:36917)
0925 10:37:42.815356 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:42.815367 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:42.815405 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:37:42.817014 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:42.817040 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:42.817648 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:42.818070 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:42.818790 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:42.820837 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:42.822247 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:42.823267 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:42.823469 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:42.824545 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:42.824557 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:42.824576 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:42.825253 (+ 677us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:42.825347 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:42.825381 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:42.825500 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:42.825677 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 10:37:42.825956 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":318,"threads_started":1}
I20260925 10:37:42.829636 22611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff: No bootstrap required, opened a new log
I20260925 10:37:42.830937 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:42.839852 22611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f1d13cda9c44c89724b74a0f9cd3ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36917 } }
I20260925 10:37:42.840323 22611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:42.840448 22611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40f1d13cda9c44c89724b74a0f9cd3ff, State: Initialized, Role: FOLLOWER
I20260925 10:37:42.841084 22611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f1d13cda9c44c89724b74a0f9cd3ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36917 } }
I20260925 10:37:42.841341 22611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:42.841475 22611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:42.841642 22611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:42.842597 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:42.835941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52277 (local address 127.0.97.62:36917)
0925 10:37:42.836135 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:42.836141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:42.836654 (+ 513us) server_negotiation.cc:407] Connection header received
0925 10:37:42.836809 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:42.836813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:42.836870 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:42.836943 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:42.837527 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:42.838285 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:42.840122 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:42.840675 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:42.840817 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:42.841889 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:42.841893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:42.841897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:42.841915 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:42.841949 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:42.841963 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:42.842062 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:42.842201 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 10:37:42.842283 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:37:42.844295 22611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f1d13cda9c44c89724b74a0f9cd3ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36917 } }
I20260925 10:37:42.844756 22611 leader_election.cc:302] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40f1d13cda9c44c89724b74a0f9cd3ff; no voters:
I20260925 10:37:42.845341 22611 leader_election.cc:288] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:42.845417 22621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:42.846191 22621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [term 1 LEADER]: Becoming Leader. State: Replica: 40f1d13cda9c44c89724b74a0f9cd3ff, State: Running, Role: LEADER
I20260925 10:37:42.846715 22621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f1d13cda9c44c89724b74a0f9cd3ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36917 } }
I20260925 10:37:42.847426 22611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:42.849408 22623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40f1d13cda9c44c89724b74a0f9cd3ff. Latest consensus state: current_term: 1 leader_uuid: "40f1d13cda9c44c89724b74a0f9cd3ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f1d13cda9c44c89724b74a0f9cd3ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36917 } } }
I20260925 10:37:42.849400 22622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40f1d13cda9c44c89724b74a0f9cd3ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f1d13cda9c44c89724b74a0f9cd3ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36917 } } }
I20260925 10:37:42.849793 22623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:42.849853 22622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:42.851012 22627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:42.857436 22627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:42.869805 22627 catalog_manager.cc:1347] Generated new cluster ID: 96aaaff7507247399e3f0520edc1bb95
I20260925 10:37:42.869920 22627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:42.888267 22627 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:42.889516 22627 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:42.895599 22627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40f1d13cda9c44c89724b74a0f9cd3ff: Generated new TSK 0
I20260925 10:37:42.896350 22627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:42.904066 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 22544
W20260925 10:37:42.905110 22606 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35387
W20260925 10:37:42.905707 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 22544: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:37:42.905856 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 22544
2026-09-25T10:37:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1854 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:43 dist-test-slave-dxdd krb5kdc[22642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:43 dist-test-slave-dxdd krb5kdc[22642](info): set up 2 sockets
Sep 25 10:37:43 dist-test-slave-dxdd krb5kdc[22642](info): commencing operation
krb5kdc: starting...
W20260925 10:37:44.002661 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:44 dist-test-slave-dxdd krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:37:44.239811 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40043
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:44.555191 22655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:44.555454 22655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:44.555538 22655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:44.562965 22655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:44.563057 22655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:44.563104 22655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:44.563159 22655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:44.563205 22655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:44.572449 22655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:44.573542 22655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:44.574702 22655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:44.586261 22660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:44.586279 22663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:44.586213 22661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:44.586984 22655 server_base.cc:1034] running on GCE node
I20260925 10:37:44.587648 22655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:44.588852 22655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:44.590057 22655 hybrid_clock.cc:648] HybridClock initialized: now 1790332664590011 us; error 25 us; skew 500 ppm
I20260925 10:37:44.590445 22655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:44 dist-test-slave-dxdd krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:44.597070 22655 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:37:44.598415 22655 webserver.cc:460] Webserver started at http://127.0.97.62:41849/ using document root <none> and password file <none>
I20260925 10:37:44.599013 22655 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:44.599112 22655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:44.599347 22655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:44.602051 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "4eb484a5e33e43f98106dd485ac06be9"
format_stamp: "Formatted at 2026-09-25 10:37:44 on dist-test-slave-dxdd"
server_key: "45cdf9f736a4df0790710f5ba88d1ac0"
server_key_iv: "b6f38bae27e11b99eb543245ab228790"
server_key_version: "encryptionkey@0"
I20260925 10:37:44.602783 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "4eb484a5e33e43f98106dd485ac06be9"
format_stamp: "Formatted at 2026-09-25 10:37:44 on dist-test-slave-dxdd"
server_key: "45cdf9f736a4df0790710f5ba88d1ac0"
server_key_iv: "b6f38bae27e11b99eb543245ab228790"
server_key_version: "encryptionkey@0"
I20260925 10:37:44.607466 22655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:37:44.610534 22670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:44.611716 22655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:37:44.611884 22655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "4eb484a5e33e43f98106dd485ac06be9"
format_stamp: "Formatted at 2026-09-25 10:37:44 on dist-test-slave-dxdd"
server_key: "45cdf9f736a4df0790710f5ba88d1ac0"
server_key_iv: "b6f38bae27e11b99eb543245ab228790"
server_key_version: "encryptionkey@0"
I20260925 10:37:44.612069 22655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:44.617628 22655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:44.621641 22655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:44.621927 22655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:44.639333 22655 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40043
I20260925 10:37:44.639400 22721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40043 every 8 connection(s)
I20260925 10:37:44.640614 22655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:44.640985 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 22655
I20260925 10:37:44.641173 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:44.641759 388 external_mini_cluster.cc:1428] Setting key 6fe7d3dd1c8ef52dba5b257182a730ea
I20260925 10:37:44.644737 22722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:44.649009 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:44.643484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55283 (local address 127.0.97.62:40043)
0925 10:37:44.644201 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:44.644212 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:44.644266 (+ 54us) server_negotiation.cc:407] Connection header received
0925 10:37:44.645857 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:44.645883 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:44.646175 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:44.646558 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:44.647554 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:44.647559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:44.647571 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:44.647589 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:44.647654 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:44.647671 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:44.647908 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:44.648038 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 10:37:44.648197 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":319,"threads_started":1}
I20260925 10:37:44.652580 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:44.655755 22722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9: Bootstrap starting.
I20260925 10:37:44.659058 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:44.656756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38303 (local address 127.0.97.62:40043)
0925 10:37:44.656934 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:44.656942 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:44.657314 (+ 372us) server_negotiation.cc:407] Connection header received
0925 10:37:44.657437 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:44.657442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:44.657504 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:44.657609 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:44.658440 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:44.658445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:44.658452 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:37:44.658469 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:44.658510 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:37:44.658529 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:44.658641 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:44.658705 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 10:37:44.658755 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:37:44.659272 22722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:44.660552 22722 log.cc:824] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:44.663149 22722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9: No bootstrap required, opened a new log
I20260925 10:37:44.670235 22722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb484a5e33e43f98106dd485ac06be9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40043 } }
I20260925 10:37:44.670605 22722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:37:44.670675 22722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb484a5e33e43f98106dd485ac06be9, State: Initialized, Role: FOLLOWER
I20260925 10:37:44.671113 22722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb484a5e33e43f98106dd485ac06be9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40043 } }
I20260925 10:37:44.671310 22722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:37:44.671411 22722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:37:44.671521 22722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:44.673358 22722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb484a5e33e43f98106dd485ac06be9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40043 } }
I20260925 10:37:44.673780 22722 leader_election.cc:302] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4eb484a5e33e43f98106dd485ac06be9; no voters:
I20260925 10:37:44.674256 22722 leader_election.cc:288] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:37:44.674423 22732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:44.675066 22732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb484a5e33e43f98106dd485ac06be9, State: Running, Role: LEADER
I20260925 10:37:44.675684 22732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb484a5e33e43f98106dd485ac06be9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40043 } }
I20260925 10:37:44.676193 22722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:44.678453 22734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4eb484a5e33e43f98106dd485ac06be9. Latest consensus state: current_term: 1 leader_uuid: "4eb484a5e33e43f98106dd485ac06be9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb484a5e33e43f98106dd485ac06be9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40043 } } }
I20260925 10:37:44.678921 22734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:44.678946 22733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4eb484a5e33e43f98106dd485ac06be9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb484a5e33e43f98106dd485ac06be9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40043 } } }
I20260925 10:37:44.679273 22733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:44.680452 22741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:44.684942 22741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:44.692759 22741 catalog_manager.cc:1347] Generated new cluster ID: cf3fbc2b589748c1a36b7c767d1c85d6
I20260925 10:37:44.692876 22741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:44.716336 22741 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:44.716805 22741 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:44.725752 22741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4eb484a5e33e43f98106dd485ac06be9: Generated new TSK 0
I20260925 10:37:44.726617 22741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:44.736600 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 22655
W20260925 10:37:44.737661 22717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55283
W20260925 10:37:44.738396 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 22655: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:37:44.738580 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 22655
2026-09-25T10:37:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1832 ms)
[----------] 18 tests from AuthTokenIssuingTest (33793 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:44 dist-test-slave-dxdd krb5kdc[22753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:44 dist-test-slave-dxdd krb5kdc[22753](info): set up 2 sockets
Sep 25 10:37:44 dist-test-slave-dxdd krb5kdc[22753](info): commencing operation
krb5kdc: starting...
W20260925 10:37:45.846796 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:46 dist-test-slave-dxdd krb5kdc[22753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:37:46.091522 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44433
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44433,127.0.97.61:34349,127.0.97.60:37035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:46.404011 22766 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:37:46.404343 22766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:46.404470 22766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:46.404613 22766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:46.411537 22766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:46.411612 22766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:46.411736 22766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:46.411839 22766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:46.411926 22766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:46.421114 22766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44433,127.0.97.61:34349,127.0.97.60:37035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:46.422322 22766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:46.423552 22766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:46.435388 22772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:46.435411 22774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:46.435518 22771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:46.436406 22766 server_base.cc:1034] running on GCE node
I20260925 10:37:46.437152 22766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:46.438504 22766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:46.439730 22766 hybrid_clock.cc:648] HybridClock initialized: now 1790332666439661 us; error 46 us; skew 500 ppm
I20260925 10:37:46.440035 22766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:46 dist-test-slave-dxdd krb5kdc[22753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:46.442945 22766 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:37:46.444196 22766 webserver.cc:460] Webserver started at http://127.0.97.62:43377/ using document root <none> and password file <none>
I20260925 10:37:46.444854 22766 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:46.444974 22766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:46.445281 22766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:46.448061 22766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "6ccc4dd565b24c889c2f11b98f70fb96"
format_stamp: "Formatted at 2026-09-25 10:37:46 on dist-test-slave-dxdd"
server_key: "aae7bd0cd0028df2041ad3a0d03c531a"
server_key_iv: "0a925ed4cdb07361be93c76453ac2829"
server_key_version: "encryptionkey@0"
I20260925 10:37:46.448805 22766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "6ccc4dd565b24c889c2f11b98f70fb96"
format_stamp: "Formatted at 2026-09-25 10:37:46 on dist-test-slave-dxdd"
server_key: "aae7bd0cd0028df2041ad3a0d03c531a"
server_key_iv: "0a925ed4cdb07361be93c76453ac2829"
server_key_version: "encryptionkey@0"
I20260925 10:37:46.453639 22766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 10:37:46.456841 22781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:46.458053 22766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:37:46.458217 22766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "6ccc4dd565b24c889c2f11b98f70fb96"
format_stamp: "Formatted at 2026-09-25 10:37:46 on dist-test-slave-dxdd"
server_key: "aae7bd0cd0028df2041ad3a0d03c531a"
server_key_iv: "0a925ed4cdb07361be93c76453ac2829"
server_key_version: "encryptionkey@0"
I20260925 10:37:46.458487 22766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:46.487207 22766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:46.491009 22766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:46.491286 22766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:46.509214 22766 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44433
I20260925 10:37:46.509241 22832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44433 every 8 connection(s)
I20260925 10:37:46.510516 22766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:46.513769 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 22766
I20260925 10:37:46.513975 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:46.513983 22833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:46.514712 388 external_mini_cluster.cc:1428] Setting key 80cd9726fa28a7d82e30f98afa167930
I20260925 10:37:46.520241 22833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:37:46 dist-test-slave-dxdd krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260925 10:37:46.544543 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:46.523386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44433 (local address 127.0.0.1:34978)
0925 10:37:46.524201 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:46.524260 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:46.526957 (+ 2697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:46.527720 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:46.527736 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:46.528496 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:46.528961 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:46.528984 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:46.530530 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:46.530538 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:46.532410 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:46.532427 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:46.533312 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:46.533322 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:46.533550 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:46.534250 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:46.534281 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:46.536858 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:46.540154 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:46.540166 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:46.540176 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:46.540750 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:46.541200 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:46.541206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:46.541210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:46.541360 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:46.542055 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:46.542067 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:46.542340 (+ 273us) client_negotiation.cc:769] Sending connection context
0925 10:37:46.542603 (+ 263us) client_negotiation.cc:240] Negotiation successful
0925 10:37:46.543309 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":187,"thread_start_us":328,"threads_started":1}
I20260925 10:37:46.544562 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:46.523386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34978 (local address 127.0.97.62:44433)
0925 10:37:46.525141 (+ 1755us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:46.525147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:46.525185 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:37:46.527287 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:46.527292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:46.527498 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:46.527644 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:46.529175 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:46.530389 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:46.532548 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:46.533078 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:46.533199 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:46.537064 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:46.537072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:46.537083 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:46.537109 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:46.540019 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:46.540870 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:46.540895 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:46.540902 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:46.541086 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:46.541476 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:46.541481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:46.541484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:46.541929 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:46.542128 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:46.542710 (+ 582us) server_negotiation.cc:299] Negotiation successful
0925 10:37:46.543152 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":1580,"spinlock_wait_cycles":14720,"thread_start_us":1386,"threads_started":1}
I20260925 10:37:46.547441 22833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:37:46.548429 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:46.548056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34349 (local address 127.0.0.1:58114)
0925 10:37:46.548212 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:46.548316 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 10:37:46.548664 22783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34349: connect: Connection refused (error 111)
I20260925 10:37:46.550127 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:46.549808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34349 (local address 127.0.0.1:58122)
0925 10:37:46.549949 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:46.550014 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 10:37:46.550329 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34349: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34349: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260925 10:37:46.573827 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34349
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44433,127.0.97.61:34349,127.0.97.60:37035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:37:46.599655 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } attempt: 1
I20260925 10:37:46.600927 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:46.600380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34349 (local address 127.0.0.1:58128)
0925 10:37:46.600617 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:46.600739 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 10:37:46.602094 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:46.601626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34349 (local address 127.0.0.1:58138)
0925 10:37:46.601786 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:46.601930 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 10:37:46.602316 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34349: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34349: connect: Connection refused (error 111)
I20260925 10:37:46.670578 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } attempt: 2
I20260925 10:37:46.671772 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:46.671253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34349 (local address 127.0.0.1:58148)
0925 10:37:46.671530 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:46.671635 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 10:37:46.672783 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:46.672459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34349 (local address 127.0.0.1:58154)
0925 10:37:46.672589 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:46.672652 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 10:37:46.673019 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34349: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34349: connect: Connection refused (error 111)
I20260925 10:37:46.764254 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } attempt: 3
I20260925 10:37:46.765322 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:46.764917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34349 (local address 127.0.0.1:58158)
0925 10:37:46.765095 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:46.765181 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 10:37:46.766283 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:46.765989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34349 (local address 127.0.0.1:58166)
0925 10:37:46.766112 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:46.766168 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:37:46.766536 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34349: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34349: connect: Connection refused (error 111)
W20260925 10:37:46.889631 22840 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:37:46.889930 22840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:46.890039 22840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:46.890180 22840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:46.897300 22840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:46.897374 22840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:46.897459 22840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:46.897548 22840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:46.897641 22840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:46.907152 22840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44433,127.0.97.61:34349,127.0.97.60:37035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:46.908299 22840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:46.909536 22840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:37:46.909765 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } attempt: 4
I20260925 10:37:46.910917 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:46.910435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34349 (local address 127.0.0.1:58170)
0925 10:37:46.910713 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:46.910795 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260925 10:37:46.911883 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:46.911574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34349 (local address 127.0.0.1:58172)
0925 10:37:46.911692 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:46.911768 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 10:37:46.912125 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34349: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34349: connect: Connection refused (error 111)
W20260925 10:37:46.921151 22845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:46.921093 22846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:46.921192 22848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:46.921846 22840 server_base.cc:1034] running on GCE node
I20260925 10:37:46.922608 22840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:46.923810 22840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:46.925041 22840 hybrid_clock.cc:648] HybridClock initialized: now 1790332666924976 us; error 43 us; skew 500 ppm
I20260925 10:37:46.925349 22840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:46 dist-test-slave-dxdd krb5kdc[22753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:46.928388 22840 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260925 10:37:46.929620 22840 webserver.cc:460] Webserver started at http://127.0.97.61:37471/ using document root <none> and password file <none>
I20260925 10:37:46.930197 22840 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:46.930285 22840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:46.930598 22840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:46.933244 22840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/data/instance:
uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e"
format_stamp: "Formatted at 2026-09-25 10:37:46 on dist-test-slave-dxdd"
server_key: "22c111e94d0f327a8c0d741b3a60510c"
server_key_iv: "7c192ae4ae87f19cc2e13bc0b1de48b3"
server_key_version: "encryptionkey@0"
I20260925 10:37:46.933914 22840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/wal/instance:
uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e"
format_stamp: "Formatted at 2026-09-25 10:37:46 on dist-test-slave-dxdd"
server_key: "22c111e94d0f327a8c0d741b3a60510c"
server_key_iv: "7c192ae4ae87f19cc2e13bc0b1de48b3"
server_key_version: "encryptionkey@0"
I20260925 10:37:46.938689 22840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 10:37:46.941715 22855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:46.942914 22840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:37:46.943075 22840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/wal
uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e"
format_stamp: "Formatted at 2026-09-25 10:37:46 on dist-test-slave-dxdd"
server_key: "22c111e94d0f327a8c0d741b3a60510c"
server_key_iv: "7c192ae4ae87f19cc2e13bc0b1de48b3"
server_key_version: "encryptionkey@0"
I20260925 10:37:46.943310 22840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:46.958432 22840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:46.962030 22840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:46.962283 22840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:46.979756 22840 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34349
I20260925 10:37:46.979787 22906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34349 every 8 connection(s)
I20260925 10:37:46.980935 22840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/data/info.pb
I20260925 10:37:46.984300 22907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:46.986755 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 22840
I20260925 10:37:46.986935 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-1/wal/instance
I20260925 10:37:46.987334 388 external_mini_cluster.cc:1428] Setting key 08eb3bc367251850a6275e31104a7b26
I20260925 10:37:46.990512 22907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:37:47 dist-test-slave-dxdd krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260925 10:37:47.011085 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:46.992904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34994 (local address 127.0.97.62:44433)
0925 10:37:46.993107 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:46.993114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:46.994160 (+ 1046us) server_negotiation.cc:407] Connection header received
0925 10:37:46.995982 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:46.995991 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:46.996080 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:46.996203 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:46.997571 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:46.998751 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.000165 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.000910 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.001053 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.004213 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:47.004222 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:47.004229 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:47.004273 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:47.008114 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.008869 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.008877 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.008882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.009036 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.009424 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.009432 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.009438 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.009838 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:47.010033 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:47.010685 (+ 652us) server_negotiation.cc:299] Negotiation successful
0925 10:37:47.010863 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:37:47.012059 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:46.993239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44433 (local address 127.0.0.1:34994)
0925 10:37:46.993981 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:46.994035 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:46.995738 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:46.996370 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:46.996377 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:46.996865 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:46.997384 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:46.997400 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:46.998893 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:46.998897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:47.000025 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:47.000036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.001103 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.001108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:47.001291 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.001929 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:47.001950 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:47.004014 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:47.008328 (+ 4314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:47.008334 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:47.008344 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:47.008753 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:47.009165 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:47.009168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:47.009171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:47.009315 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:47.009962 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:47.009967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:47.010242 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 10:37:47.010560 (+ 318us) client_negotiation.cc:240] Negotiation successful
0925 10:37:47.010900 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":265,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260925 10:37:47.013806 22907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:37:47 dist-test-slave-dxdd krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260925 10:37:47.027691 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.014657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34349 (local address 127.0.0.1:58180)
0925 10:37:47.014811 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.014825 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:47.014902 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:47.015851 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:47.015855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:47.016245 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:47.016352 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:47.016359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.017726 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.017732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:47.018601 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:47.018612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.019400 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.019406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:47.019671 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.020222 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:47.020236 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:47.022413 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:47.025401 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:47.025409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:47.025413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:47.025747 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:47.026082 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:47.026088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:47.026091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:47.026213 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:47.026790 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:47.026794 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:47.026916 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 10:37:47.027066 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 10:37:47.027319 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":55}
I20260925 10:37:47.027786 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.014933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.0.97.61:34349)
0925 10:37:47.015484 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.015489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.015512 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:37:47.015555 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.015560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.015710 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.015861 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:47.016543 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.017585 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.018750 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.019266 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.019404 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.022635 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:47.022645 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:47.022655 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:47.022684 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:47.025282 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.025867 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.025881 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.025883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.025981 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.026323 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.026330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.026335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.026648 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:47.026850 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:47.027189 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 10:37:47.027414 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":429,"thread_start_us":304,"threads_started":1}
I20260925 10:37:47.030203 22907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:37:47.031302 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.030881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48030)
0925 10:37:47.031050 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.031166 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 10:37:47.031541 22856 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
I20260925 10:37:47.033075 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.032735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48042)
0925 10:37:47.032879 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.032957 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 10:37:47.033295 22907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37035: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260925 10:37:47.041510 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37035
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44433,127.0.97.61:34349,127.0.97.60:37035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:37:47.082561 22907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } attempt: 1
I20260925 10:37:47.083703 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.083284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48044)
0925 10:37:47.083469 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.083559 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 10:37:47.084719 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.084416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48056)
0925 10:37:47.084558 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.084620 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 10:37:47.084991 22907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37035: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
I20260925 10:37:47.153208 22907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } attempt: 2
I20260925 10:37:47.154392 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.153908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48070)
0925 10:37:47.154087 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.154192 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 10:37:47.155318 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.154993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48072)
0925 10:37:47.155131 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.155211 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 10:37:47.155505 22907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37035: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
I20260925 10:37:47.211396 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } attempt: 5
Sep 25 10:37:47 dist-test-slave-dxdd krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260925 10:37:47.224684 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.212309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.0.97.61:34349)
0925 10:37:47.212497 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.212503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.212524 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:37:47.212651 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.212657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.212740 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.212843 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:47.213584 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.214430 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.215547 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.215949 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.216076 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.219651 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:47.219656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:47.219660 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:47.219689 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:47.222172 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.223073 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.223077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.223079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.223201 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.223570 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.223573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.223575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.223809 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:47.223910 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:47.224326 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 10:37:47.224475 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:37:47.224961 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.212154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34349 (local address 127.0.0.1:58190)
0925 10:37:47.212358 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.212394 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:47.212524 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:47.212884 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:47.212891 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:47.213267 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:47.213438 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:47.213446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.214566 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.214577 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:47.215402 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:47.215413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.216086 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.216091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:47.216313 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.216993 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:47.217011 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:47.219456 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:47.222374 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:47.222382 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:47.222387 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:47.222964 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:47.223308 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:47.223312 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:47.223316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:47.223472 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:47.223920 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:47.223928 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:47.224098 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 10:37:47.224246 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 10:37:47.224502 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 10:37:47.225886 22833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:37:47.226884 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.226505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48086)
0925 10:37:47.226665 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.226733 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 10:37:47.227741 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.227443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48092)
0925 10:37:47.227591 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.227652 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 10:37:47.227952 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37035: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
I20260925 10:37:47.246709 22907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } attempt: 3
I20260925 10:37:47.247721 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.247292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48108)
0925 10:37:47.247487 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.247575 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 10:37:47.248728 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.248350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48110)
0925 10:37:47.248560 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.248631 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260925 10:37:47.249010 22907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37035: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
I20260925 10:37:47.279239 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } attempt: 1
I20260925 10:37:47.280524 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.279938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48118)
0925 10:37:47.280163 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.280275 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 10:37:47.281782 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.281254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48120)
0925 10:37:47.281511 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.281621 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 10:37:47.282135 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37035: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
I20260925 10:37:47.350378 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } attempt: 2
I20260925 10:37:47.351541 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.351078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48134)
0925 10:37:47.351300 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.351387 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 10:37:47.352553 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.352245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48140)
0925 10:37:47.352378 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.352446 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 10:37:47.352849 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37035: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
W20260925 10:37:47.369573 22914 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:37:47.369849 22914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:47.369961 22914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:47.370121 22914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:47.377287 22914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:47.377364 22914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:47.377493 22914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:47.377571 22914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:47.377694 22914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:47.387463 22914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44433,127.0.97.61:34349,127.0.97.60:37035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:47.388701 22914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:47.389981 22914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:37:47.392206 22907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } attempt: 4
I20260925 10:37:47.393287 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.392820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48146)
0925 10:37:47.393048 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.393135 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 10:37:47.394182 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.393878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48150)
0925 10:37:47.394016 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.394085 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:37:47.394425 22907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37035: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
W20260925 10:37:47.401530 22919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:47.401516 22920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:47.401541 22922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:47.402493 22914 server_base.cc:1034] running on GCE node
I20260925 10:37:47.403162 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:47.404371 22914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:47.405577 22914 hybrid_clock.cc:648] HybridClock initialized: now 1790332667405529 us; error 28 us; skew 500 ppm
I20260925 10:37:47.405838 22914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:47 dist-test-slave-dxdd krb5kdc[22753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:47.408809 22914 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260925 10:37:47.410185 22914 webserver.cc:460] Webserver started at http://127.0.97.60:44281/ using document root <none> and password file <none>
I20260925 10:37:47.410851 22914 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:47.410933 22914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:47.411198 22914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:47.413924 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/data/instance:
uuid: "03156faf66f8499eb86443b156c4e088"
format_stamp: "Formatted at 2026-09-25 10:37:47 on dist-test-slave-dxdd"
server_key: "f4af9ee7b33d7bec1c5f18f1bf3f7367"
server_key_iv: "b5616673d789f14f0db454f62281ace5"
server_key_version: "encryptionkey@0"
I20260925 10:37:47.414702 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/wal/instance:
uuid: "03156faf66f8499eb86443b156c4e088"
format_stamp: "Formatted at 2026-09-25 10:37:47 on dist-test-slave-dxdd"
server_key: "f4af9ee7b33d7bec1c5f18f1bf3f7367"
server_key_iv: "b5616673d789f14f0db454f62281ace5"
server_key_version: "encryptionkey@0"
I20260925 10:37:47.419334 22914 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:37:47.422423 22929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:47.423635 22914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:37:47.423799 22914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/wal
uuid: "03156faf66f8499eb86443b156c4e088"
format_stamp: "Formatted at 2026-09-25 10:37:47 on dist-test-slave-dxdd"
server_key: "f4af9ee7b33d7bec1c5f18f1bf3f7367"
server_key_iv: "b5616673d789f14f0db454f62281ace5"
server_key_version: "encryptionkey@0"
I20260925 10:37:47.423996 22914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:47.438398 22914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:47.442553 22914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:47.442800 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:47.459053 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } attempt: 3
I20260925 10:37:47.460269 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.459689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48152)
0925 10:37:47.459962 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.460084 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 10:37:47.461018 22914 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37035
I20260925 10:37:47.461038 22980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37035 every 8 connection(s)
I20260925 10:37:47.462486 22914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/data/info.pb
I20260925 10:37:47.466491 22982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 10:37:47 dist-test-slave-dxdd krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260925 10:37:47.472896 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 22914
I20260925 10:37:47.473088 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332315635355-388-0/minicluster-data/master-2/wal/instance
I20260925 10:37:47.473573 388 external_mini_cluster.cc:1428] Setting key de85b4cd991751c6367532db9515594d
I20260925 10:37:47.475903 22982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:37:47.481462 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.460918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48162)
0925 10:37:47.461127 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.461145 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:47.461324 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:47.465553 (+ 4229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:47.465558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:47.466050 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:47.466252 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:47.466268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.467868 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.467876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:47.469235 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:47.469249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.470125 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.470130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:47.470329 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.470884 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:47.470906 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:47.473435 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:47.478345 (+ 4910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:47.478353 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:47.478361 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:47.479001 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:47.479532 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:47.479538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:47.479542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:47.479688 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:47.480536 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:47.480540 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:47.480652 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 10:37:47.480787 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 10:37:47.481081 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 10:37:47.482905 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.462113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48162 (local address 127.0.97.60:37035)
0925 10:37:47.463050 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.463065 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.463127 (+ 62us) server_negotiation.cc:407] Connection header received
0925 10:37:47.464889 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.464909 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.465251 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.465545 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:47.466428 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.467725 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.469364 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.469986 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.470120 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.473695 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:47.473706 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:47.473723 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:47.473778 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:47.478150 (+ 4372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.479173 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.479180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.479187 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.479384 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.479824 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.479829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.479833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.480420 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:47.480589 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:47.481058 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 10:37:47.481574 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":464,"threads_started":1}
Sep 25 10:37:47 dist-test-slave-dxdd krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 25 10:37:47 dist-test-slave-dxdd krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:37:47.489965 22833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96: Bootstrap starting.
I20260925 10:37:47.492636 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.474557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.0.97.60:37035)
0925 10:37:47.475538 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.475547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.475568 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:37:47.475614 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.475620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.475699 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.475811 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:47.477865 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.479254 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.481039 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.481754 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.481890 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.486579 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:47.486585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:47.486590 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:47.486624 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:47.489797 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.490616 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.490624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.490628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.490755 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.491131 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.491135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.491139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.491486 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:47.491758 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:47.492045 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 10:37:47.492249 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":633,"threads_started":1}
I20260925 10:37:47.494719 22833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:47.496188 22833 log.cc:824] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:47.496781 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.478469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44433 (local address 127.0.0.1:35010)
0925 10:37:47.479220 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.479264 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:47.479457 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:47.480664 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:47.480670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:47.481540 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:47.481834 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:47.481847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.483023 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.483028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:47.484495 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:47.484506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.485485 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.485491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:47.485699 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.486298 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:47.486364 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:47.489292 (+ 2928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:47.493205 (+ 3913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:47.493216 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:47.493223 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:47.493779 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:47.494732 (+ 953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:47.494740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:47.494743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:47.494922 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:47.495587 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:47.495594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:47.495801 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 10:37:47.496002 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 10:37:47.496317 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":388,"threads_started":1}
I20260925 10:37:47.497999 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.478459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.0.97.62:44433)
0925 10:37:47.478682 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.478691 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.479346 (+ 655us) server_negotiation.cc:407] Connection header received
0925 10:37:47.479606 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.479613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.479685 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.479798 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:47.481988 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.482867 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.484639 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.485297 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.485433 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.489502 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:47.489510 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:47.489515 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:47.489553 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:47.493067 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.493907 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.493914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.493919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.494553 (+ 634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.495054 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.495065 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.495071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.495445 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:47.495597 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:47.497386 (+ 1789us) server_negotiation.cc:299] Negotiation successful
0925 10:37:47.497612 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:37:47.499269 22982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:37:47.499390 22833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96: No bootstrap required, opened a new log
Sep 25 10:37:47 dist-test-slave-dxdd krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 25 10:37:47 dist-test-slave-dxdd krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260925 10:37:47.510279 22833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } }
I20260925 10:37:47.510727 22833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ccc4dd565b24c889c2f11b98f70fb96, State: Initialized, Role: FOLLOWER
I20260925 10:37:47.511314 22833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } }
I20260925 10:37:47.512079 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.494379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35024 (local address 127.0.97.62:44433)
0925 10:37:47.495483 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.495488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.495509 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:37:47.495554 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.495559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.495620 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.495735 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:47.497259 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.498273 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.499707 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.500349 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.500460 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.504872 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:47.504878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:47.504883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:47.504921 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:47.508644 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.509524 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.509529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.509533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.509705 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.510140 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.510144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.510147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.510534 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:47.510797 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:47.511369 (+ 572us) server_negotiation.cc:299] Negotiation successful
0925 10:37:47.511586 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":988,"thread_start_us":773,"threads_started":1}
I20260925 10:37:47.512573 22989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } } }
I20260925 10:37:47.513031 22989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:37:47.513899 22833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:47.524855 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.500085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34349 (local address 127.0.0.1:58202)
0925 10:37:47.500327 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.500343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:47.500443 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:47.500772 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:47.500777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:47.501496 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:47.501679 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:47.501692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.503175 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.503181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:47.504518 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:47.504532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.505321 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.505327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:47.505505 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.506072 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:47.506089 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:47.508040 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:47.522426 (+ 14386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:47.522431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:47.522434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:47.522862 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:47.523312 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:47.523318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:47.523323 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:47.523501 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:47.524082 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:47.524087 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:47.524265 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 10:37:47.524415 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 10:37:47.524639 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
Sep 25 10:37:47 dist-test-slave-dxdd krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260925 10:37:47.525813 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.500063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.0.97.61:34349)
0925 10:37:47.500241 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.500247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.500423 (+ 176us) server_negotiation.cc:407] Connection header received
0925 10:37:47.500544 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.500550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.500631 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.500729 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:47.501846 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.503011 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.504684 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.505155 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.505310 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.508230 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:47.508234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:47.508237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:47.508267 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:47.511901 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.522977 (+ 11076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.522984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.522989 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.523153 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.523645 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.523650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.523666 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.523969 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:47.524115 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:47.525245 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0925 10:37:47.525448 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:37:47.527128 22982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:37:47.533353 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.514464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.0.97.61:34349)
0925 10:37:47.514976 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.514982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.515002 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:37:47.515043 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.515048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.515105 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.516597 (+ 1492us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:47.516827 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.517944 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.520245 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.520872 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.520992 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.526680 (+ 5688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:47.526685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:47.526689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:47.526720 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:47.530797 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.531467 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.531471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.531473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.531588 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.531956 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.531961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.531964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.532317 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:47.532581 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:47.532934 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 10:37:47.533085 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":236,"threads_started":1}
I20260925 10:37:47.533888 23001 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
Sep 25 10:37:47 dist-test-slave-dxdd krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260925 10:37:47.541791 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.527787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48184)
0925 10:37:47.527926 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.527943 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:47.528026 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:47.528435 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:47.528438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:47.528877 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:47.529028 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:47.529036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.529938 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.529946 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:47.530838 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:47.530848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.532108 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.532114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:47.532320 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.533703 (+ 1383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:47.533719 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:47.536176 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:47.539381 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:47.539386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:47.539390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:47.539889 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:47.540203 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:47.540207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:47.540211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:47.540360 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:47.540816 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:47.540820 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:47.540969 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 10:37:47.541113 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 10:37:47.541427 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 10:37:47.541961 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.528004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.0.97.60:37035)
0925 10:37:47.528146 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.528152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.528179 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:37:47.528224 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.528228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.528311 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.528401 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:47.529162 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.529823 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.531012 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.531939 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.532097 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.536341 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:47.536346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:47.536351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:47.536387 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:47.539274 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.539995 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.539999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.540001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.540106 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.540457 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.540460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.540462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.540708 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:47.540813 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:47.541218 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 10:37:47.541625 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":38}
I20260925 10:37:47.542294 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:47.545604 22808 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6ccc4dd565b24c889c2f11b98f70fb96"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35024
I20260925 10:37:47.546027 22808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 10:37:47.546221 22808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:47.538558 22766 master_runner.cc:186] Error getting master registration for 127.0.97.61:34349: OK, instance_id { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" instance_seqno: 1790332666974795 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:37:47.547097 22766 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:37:47.547242 22982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088: Bootstrap starting.
I20260925 10:37:47.548715 22808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } }
I20260925 10:37:47.549804 22808 leader_election.cc:288] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [CANDIDATE]: Term 1 election: Requested vote from peers 33a5d02ccdd34cdb92e8532e4ed6b64e (127.0.97.61:34349), 03156faf66f8499eb86443b156c4e088 (127.0.97.60:37035)
I20260925 10:37:47.550458 22982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:47.551738 22982 log.cc:824] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:47.555190 22982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088: No bootstrap required, opened a new log
I20260925 10:37:47.568440 22982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } }
I20260925 10:37:47.568897 22982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03156faf66f8499eb86443b156c4e088, State: Initialized, Role: FOLLOWER
I20260925 10:37:47.569394 22982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } }
I20260925 10:37:47.574223 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.554906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35034 (local address 127.0.97.62:44433)
0925 10:37:47.555077 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.555084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.555208 (+ 124us) server_negotiation.cc:407] Connection header received
0925 10:37:47.555465 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.555470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.555562 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.555658 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:47.558514 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.559904 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.562509 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.563372 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.563485 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.567492 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:47.567500 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:47.567506 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:47.567548 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:47.570985 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.571825 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.571832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.571838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.571983 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.572441 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.572446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.572449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.572780 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:47.573033 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:47.573604 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 10:37:47.573825 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:37:47.579828 23010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } } }
I20260925 10:37:47.580866 23010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:37:47.581271 22982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:47.581369 22956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "03156faf66f8499eb86443b156c4e088"
I20260925 10:37:47.581813 22956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:47.585018 22956 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ccc4dd565b24c889c2f11b98f70fb96 in term 1.
I20260925 10:37:47.585783 22782 leader_election.cc:302] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03156faf66f8499eb86443b156c4e088, 6ccc4dd565b24c889c2f11b98f70fb96; no voters:
I20260925 10:37:47.586390 22989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:47.587831 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.554452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.0.97.61:34349)
0925 10:37:47.554638 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.554645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.564977 (+ 10332us) server_negotiation.cc:407] Connection header received
0925 10:37:47.565306 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.565311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.565401 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.565472 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:47.567763 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.568842 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.571264 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.571947 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.572048 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.576062 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:47.576068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:47.576072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:47.576111 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:47.579601 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.580767 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.581031 (+ 264us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.581036 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.581210 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.585918 (+ 4708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.585923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.585928 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.586320 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:47.586558 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:47.587408 (+ 850us) server_negotiation.cc:299] Negotiation successful
0925 10:37:47.587615 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:37:47.593000 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.555869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.0.97.60:37035)
0925 10:37:47.556046 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.556053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.569196 (+ 13143us) server_negotiation.cc:407] Connection header received
0925 10:37:47.569344 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.569348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.569432 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.569528 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:47.575313 (+ 5785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.576765 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.579334 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.580105 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.580247 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.583260 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:47.583274 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:47.583281 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:47.583334 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:47.587348 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.590565 (+ 3217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.590571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.590574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.590717 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.591160 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.591166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.591169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.591628 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:47.591906 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:47.592216 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 10:37:47.592472 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 10:37:47.601171 23022 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:37:47.601346 23022 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:37:47.604213 22989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [term 1 LEADER]: Becoming Leader. State: Replica: 6ccc4dd565b24c889c2f11b98f70fb96, State: Running, Role: LEADER
I20260925 10:37:47.604743 22989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } }
I20260925 10:37:47.609455 23014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ccc4dd565b24c889c2f11b98f70fb96. Latest consensus state: current_term: 1 leader_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } } }
I20260925 10:37:47.609848 23014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:47.610616 23025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:47.616077 23025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:47.612833 22914 master_runner.cc:186] Error getting master registration for 127.0.97.61:34349: OK, instance_id { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" instance_seqno: 1790332666974795 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:37:47.628002 22914 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:37:47.630427 22956 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088 [term 1 FOLLOWER]: Refusing update from remote peer 6ccc4dd565b24c889c2f11b98f70fb96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:37:47.631937 22989 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:37:47.651853 23010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ccc4dd565b24c889c2f11b98f70fb96. Latest consensus state: current_term: 1 leader_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } } }
I20260925 10:37:47.652153 23010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:37:47.653353 23027 mvcc.cc:205] Tried to move back new op lower bound from 7333202606605119488 to 7333202606513852416. Current Snapshot: MvccSnapshot[applied={T|T < 7333202606605119488}]
I20260925 10:37:47.666563 22989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } } }
I20260925 10:37:47.666858 22989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:47.668030 23025 catalog_manager.cc:1347] Generated new cluster ID: cdc505f6012e48888879f498b54dab29
I20260925 10:37:47.668149 23025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:47.670331 23010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } } }
I20260925 10:37:47.670571 23010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:37:47.693655 22907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } attempt: 5
I20260925 10:37:47.703684 23025 catalog_manager.cc:1370] Generated new certificate authority record
Sep 25 10:37:47 dist-test-slave-dxdd krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260925 10:37:47.705565 23025 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:47.712567 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.694180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37035 (local address 127.0.0.1:48200)
0925 10:37:47.694476 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:47.694498 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:47.694601 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:47.696232 (+ 1631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:47.696238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:47.696715 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:47.696863 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:47.696872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.698331 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.698337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:47.699716 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:47.699731 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.700926 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.700932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:47.701116 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.701852 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:47.702585 (+ 733us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:47.705463 (+ 2878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:47.709844 (+ 4381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:47.709850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:47.709853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:47.710370 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:47.710789 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:47.710794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:47.710798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:47.710947 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:47.711600 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:47.711606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:47.711756 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 10:37:47.711887 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 10:37:47.712166 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260925 10:37:47.713450 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.694857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48200 (local address 127.0.97.60:37035)
0925 10:37:47.695001 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.695005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.695021 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:37:47.695068 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.695072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.695140 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.695252 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:47.697023 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.698145 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.699897 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.700604 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.700734 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.705683 (+ 4949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:47.705690 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:47.705694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:47.705731 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:47.709687 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.710501 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.710506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.710509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.710665 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:47.711071 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:47.711078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:47.711082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:47.711412 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:47.711543 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:47.712880 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0925 10:37:47.713095 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:37:47.718873 22907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e: Bootstrap starting.
I20260925 10:37:47.723188 22907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:47.724391 22907 log.cc:824] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:47.734277 23025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96: Generated new TSK 0
I20260925 10:37:47.735486 23025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:37:47.738859 22907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e: No bootstrap required, opened a new log
I20260925 10:37:47.750415 22907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } }
I20260925 10:37:47.750780 22907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a5d02ccdd34cdb92e8532e4ed6b64e, State: Initialized, Role: FOLLOWER
I20260925 10:37:47.751305 22907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } }
I20260925 10:37:47.752637 23033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } } }
I20260925 10:37:47.753422 23033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:37:47.753752 22907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:47.758507 22881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:47.762431 22882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e"
I20260925 10:37:47.765799 22881 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e [term 1 FOLLOWER]: Refusing update from remote peer 6ccc4dd565b24c889c2f11b98f70fb96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:37:47.769886 22882 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ccc4dd565b24c889c2f11b98f70fb96 in term 1.
I20260925 10:37:47.776552 22989 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:37:47.843556 23033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ccc4dd565b24c889c2f11b98f70fb96. Latest consensus state: current_term: 1 leader_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } } }
I20260925 10:37:47.843993 23033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:37:47.844398 23040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } } }
I20260925 10:37:47.844837 23040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:37:47.881209 23051 catalog_manager.cc:1259] Loaded cluster ID: cdc505f6012e48888879f498b54dab29
I20260925 10:37:47.881319 23051 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e: loading cluster ID for follower catalog manager: success
I20260925 10:37:47.892577 23051 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e: acquiring CA information for follower catalog manager: success
I20260925 10:37:47.896792 23051 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 33a5d02ccdd34cdb92e8532e4ed6b64e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 10:37:47.920735 22987 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:47.912442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.0.97.62:44433)
0925 10:37:47.912685 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.912695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.918802 (+ 6107us) server_negotiation.cc:407] Connection header received
0925 10:37:47.919094 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.919100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.919201 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.919282 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:47.920468 (+ 1186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35036: BlockingRecv error: recv got EOF from 127.0.0.1:35036 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260925 10:37:47.922823 22990 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:47.918560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.0.97.61:34349)
0925 10:37:47.918703 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.918709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.921229 (+ 2520us) server_negotiation.cc:407] Connection header received
0925 10:37:47.921461 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.921466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.921553 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.921621 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:47.922571 (+ 950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58228: BlockingRecv error: recv got EOF from 127.0.0.1:58228 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 10:37:47.925074 22985 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:47.918201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.0.97.60:37035)
0925 10:37:47.918410 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.918417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.923593 (+ 5176us) server_negotiation.cc:407] Connection header received
0925 10:37:47.923837 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.923842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.923917 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.923988 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:47.924831 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48212: BlockingRecv error: recv got EOF from 127.0.0.1:48212 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:37:47.973408 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:47.958335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.0.97.61:34349)
0925 10:37:47.958574 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:47.958584 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:47.966268 (+ 7684us) server_negotiation.cc:407] Connection header received
0925 10:37:47.966607 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:47.966614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:47.966841 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:47.966965 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:47.967655 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.968743 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.970658 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:47.971297 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:47.971383 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:47.972026 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:47.972653 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:47.972771 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:47.972999 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 10:37:47.973087 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 10:37:48.037047 22985 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:48.018406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.0.97.60:37035)
0925 10:37:48.018594 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.018602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.035118 (+ 16516us) server_negotiation.cc:407] Connection header received
0925 10:37:48.035456 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.035463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.035595 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.035694 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:48.036804 (+ 1110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48218: BlockingRecv error: recv got EOF from 127.0.0.1:48218 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:37:48.091341 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:48.073573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58242 (local address 127.0.97.61:34349)
0925 10:37:48.073752 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.073758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.084288 (+ 10530us) server_negotiation.cc:407] Connection header received
0925 10:37:48.084820 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.084826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.084912 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.084980 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:48.085688 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.086839 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.088867 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.089617 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.089738 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:48.090413 (+ 675us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:48.090575 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:48.090640 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:48.090874 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 10:37:48.090974 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 10:37:48.137461 22985 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:48.124357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.0.97.60:37035)
0925 10:37:48.124550 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.124557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.135323 (+ 10766us) server_negotiation.cc:407] Connection header received
0925 10:37:48.135855 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.135861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.135982 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.136076 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:48.137192 (+ 1116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48228: BlockingRecv error: recv got EOF from 127.0.0.1:48228 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:37:48.192106 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:48.174385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.0.97.61:34349)
0925 10:37:48.174620 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.174628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.185192 (+ 10564us) server_negotiation.cc:407] Connection header received
0925 10:37:48.185700 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.185708 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.185812 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.185890 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:48.186587 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.187671 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.189734 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.190451 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.190551 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:48.191205 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:48.191376 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:48.191441 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:48.191671 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 10:37:48.191761 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 10:37:48.198608 22989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ccc4dd565b24c889c2f11b98f70fb96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } peers { permanent_uuid: "33a5d02ccdd34cdb92e8532e4ed6b64e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34349 } } peers { permanent_uuid: "03156faf66f8499eb86443b156c4e088" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37035 } } }
I20260925 10:37:48.199012 22989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ccc4dd565b24c889c2f11b98f70fb96 [sys.catalog]: This master's current role is: LEADER
W20260925 10:37:48.231415 22985 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:48.224758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48234 (local address 127.0.97.60:37035)
0925 10:37:48.224929 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.224936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.229832 (+ 4896us) server_negotiation.cc:407] Connection header received
0925 10:37:48.230034 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.230040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.230147 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.230275 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:48.231189 (+ 914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48234: BlockingRecv error: recv got EOF from 127.0.0.1:48234 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:37:48.249140 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:48.238460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.0.97.61:34349)
0925 10:37:48.238700 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.238707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.242841 (+ 4134us) server_negotiation.cc:407] Connection header received
0925 10:37:48.243076 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.243082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.243171 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.243277 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:48.243714 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.244932 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.247025 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.247790 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.247941 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:48.248376 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:48.248533 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:48.248627 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:48.248790 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 10:37:48.248866 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260925 10:37:48.262286 22985 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:48.260025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48250 (local address 127.0.97.60:37035)
0925 10:37:48.260217 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.260222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.260681 (+ 459us) server_negotiation.cc:407] Connection header received
0925 10:37:48.260915 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.260920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.261014 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.261085 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:48.262065 (+ 980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48250: BlockingRecv error: recv got EOF from 127.0.0.1:48250 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 10:37:48.279521 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:48.273776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.0.97.61:34349)
0925 10:37:48.273965 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.273970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.274278 (+ 308us) server_negotiation.cc:407] Connection header received
0925 10:37:48.274534 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.274538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.274615 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.274691 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:48.275155 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.275859 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.277812 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.278274 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.278433 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:48.278838 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:48.278932 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:48.278993 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:48.279131 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 10:37:48.279214 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 10:37:48.286935 22985 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:48.285155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48262 (local address 127.0.97.60:37035)
0925 10:37:48.285352 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.285359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.285648 (+ 289us) server_negotiation.cc:407] Connection header received
0925 10:37:48.285757 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.285762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.285869 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.285998 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:48.286715 (+ 717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48262: BlockingRecv error: recv got EOF from 127.0.0.1:48262 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:37:48.311626 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:48.305720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.0.97.61:34349)
0925 10:37:48.305941 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.305947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.306286 (+ 339us) server_negotiation.cc:407] Connection header received
0925 10:37:48.306523 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.306529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.306613 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.306716 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:48.307120 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.308259 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.309662 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.310383 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.310518 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:48.310870 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:48.311022 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:48.311127 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:48.311240 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 10:37:48.311325 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 10:37:48.319272 22985 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:48.317337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.0.97.60:37035)
0925 10:37:48.317524 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.317533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.317903 (+ 370us) server_negotiation.cc:407] Connection header received
0925 10:37:48.318144 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.318149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.318246 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.318391 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:48.319027 (+ 636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48270: BlockingRecv error: recv got EOF from 127.0.0.1:48270 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:37:48.360324 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:48.354397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.0.97.61:34349)
0925 10:37:48.354623 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.354630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.355067 (+ 437us) server_negotiation.cc:407] Connection header received
0925 10:37:48.355286 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.355291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.355364 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.355439 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:48.355892 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.356682 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.358665 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.359065 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.359152 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:48.359613 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:48.359740 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:48.359798 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:48.359969 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 10:37:48.360045 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 10:37:48.367328 22985 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:48.365655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48276 (local address 127.0.97.60:37035)
0925 10:37:48.365831 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.365836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.366119 (+ 283us) server_negotiation.cc:407] Connection header received
0925 10:37:48.366217 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.366224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.366344 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.366448 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:48.367116 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48276: BlockingRecv error: recv got EOF from 127.0.0.1:48276 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:37:48.438961 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:48.434404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.0.97.61:34349)
0925 10:37:48.434617 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.434623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.434964 (+ 341us) server_negotiation.cc:407] Connection header received
0925 10:37:48.435113 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.435117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.435178 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.435252 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:48.435557 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.436254 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.437514 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.437890 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.437976 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:48.438285 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:48.438453 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:48.438547 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:48.438695 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 10:37:48.438752 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 10:37:48.445873 22985 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:48.444265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.0.97.60:37035)
0925 10:37:48.444472 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.444480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.444666 (+ 186us) server_negotiation.cc:407] Connection header received
0925 10:37:48.444798 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.444802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.444865 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.444941 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:48.445624 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48280: BlockingRecv error: recv got EOF from 127.0.0.1:48280 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:37:48.581688 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:48.577116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58316 (local address 127.0.97.61:34349)
0925 10:37:48.577354 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.577360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.577612 (+ 252us) server_negotiation.cc:407] Connection header received
0925 10:37:48.577783 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.577787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.577856 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.577935 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:48.578321 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.579003 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.580173 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.580591 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.580677 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:48.581009 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:48.581105 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:48.581194 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:48.581311 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 10:37:48.581367 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260925 10:37:48.588894 22985 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:48.587118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.0.97.60:37035)
0925 10:37:48.587347 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.587354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.587571 (+ 217us) server_negotiation.cc:407] Connection header received
0925 10:37:48.587777 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.587781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.587845 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.587945 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:48.588676 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48296: BlockingRecv error: recv got EOF from 127.0.0.1:48296 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:37:48.603164 23022 catalog_manager.cc:1259] Loaded cluster ID: cdc505f6012e48888879f498b54dab29
I20260925 10:37:48.603240 23022 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088: loading cluster ID for follower catalog manager: success
I20260925 10:37:48.605566 23022 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088: acquiring CA information for follower catalog manager: success
I20260925 10:37:48.607360 23022 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 03156faf66f8499eb86443b156c4e088: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:37:48.853545 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:48.848495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58326 (local address 127.0.97.61:34349)
0925 10:37:48.848722 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.848729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.849009 (+ 280us) server_negotiation.cc:407] Connection header received
0925 10:37:48.849261 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.849267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.849347 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.849442 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:48.849846 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.850629 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.851932 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.852351 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.852480 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:48.852840 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:48.852934 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:48.853046 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:48.853193 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 10:37:48.853268 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:37:48.863679 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:48.858646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.0.97.60:37035)
0925 10:37:48.858847 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.858852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.859046 (+ 194us) server_negotiation.cc:407] Connection header received
0925 10:37:48.859164 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.859167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.859303 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.859391 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:48.859730 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.860388 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.861707 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.862069 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.862157 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:48.862631 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:48.863128 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:48.863249 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:48.863305 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 10:37:48.863358 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:37:48.876737 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:48.870289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.0.97.60:37035)
0925 10:37:48.870455 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.870463 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.870992 (+ 529us) server_negotiation.cc:407] Connection header received
0925 10:37:48.871199 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.871204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.871271 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.871362 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:48.872003 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.872917 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.874681 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.875217 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.875336 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:48.875721 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:48.875833 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:48.875914 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:48.876406 (+ 492us) server_negotiation.cc:299] Negotiation successful
0925 10:37:48.876470 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:37:48.876720 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:48.869195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58336 (local address 127.0.97.61:34349)
0925 10:37:48.869378 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.869386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.870383 (+ 997us) server_negotiation.cc:407] Connection header received
0925 10:37:48.870643 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.870651 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.870736 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.870800 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:48.871613 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.872473 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.874887 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.875497 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.875593 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:48.875997 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:48.876107 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:48.876203 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:48.876372 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 10:37:48.876452 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:37:48.878242 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:48.869139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.0.97.62:44433)
0925 10:37:48.869802 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:48.869811 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:48.870205 (+ 394us) server_negotiation.cc:407] Connection header received
0925 10:37:48.870401 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:48.870409 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:48.870624 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:48.871468 (+ 844us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:48.871647 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.872619 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.875693 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:48.876191 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:48.876318 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:48.876777 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:48.877563 (+ 786us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:48.877699 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:48.877843 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 10:37:48.877917 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":365,"threads_started":1}
I20260925 10:37:48.882783 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 22766
I20260925 10:37:48.883832 22828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:48.990856 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 22766
I20260925 10:37:49.011823 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 22840
I20260925 10:37:49.012912 22902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:49.122804 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 22840
I20260925 10:37:49.142654 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 22914
I20260925 10:37:49.143689 22976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:49.248399 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 22914
2026-09-25T10:37:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4514 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:37:49 dist-test-slave-dxdd krb5kdc[23174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:37:49 dist-test-slave-dxdd krb5kdc[23174](info): set up 2 sockets
Sep 25 10:37:49 dist-test-slave-dxdd krb5kdc[23174](info): commencing operation
krb5kdc: starting...
W20260925 10:37:50.342801 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/test-user.keytab.
Sep 25 10:37:50 dist-test-slave-dxdd krb5kdc[23174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:37:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:37:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260925 10:37:50.579409 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36507
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36507,127.0.97.61:40753,127.0.97.60:33847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:37:50.901867 23187 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:37:50.902226 23187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:50.902401 23187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:50.902534 23187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:50.909665 23187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:50.909742 23187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:50.909819 23187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:50.909896 23187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:50.910009 23187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:50.919718 23187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36507,127.0.97.61:40753,127.0.97.60:33847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:50.921206 23187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:50.922524 23187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:50.933625 23192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:50.933761 23195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:50.933653 23193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:50.934403 23187 server_base.cc:1034] running on GCE node
I20260925 10:37:50.935315 23187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:50.936887 23187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:50.938112 23187 hybrid_clock.cc:648] HybridClock initialized: now 1790332670938044 us; error 38 us; skew 500 ppm
I20260925 10:37:50.938504 23187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:50 dist-test-slave-dxdd krb5kdc[23174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:50.947090 23187 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260925 10:37:50.948653 23187 webserver.cc:460] Webserver started at http://127.0.97.62:35919/ using document root <none> and password file <none>
I20260925 10:37:50.949350 23187 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:50.949465 23187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:50.949795 23187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:50.952751 23187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/data/instance:
uuid: "258a4522d3da4bdcba1e46b22bed40a4"
format_stamp: "Formatted at 2026-09-25 10:37:50 on dist-test-slave-dxdd"
server_key: "acd5516eb27caeb947779d8bad527a24"
server_key_iv: "33888134fc19131476e3b35410cfeb08"
server_key_version: "encryptionkey@0"
I20260925 10:37:50.953529 23187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/wal/instance:
uuid: "258a4522d3da4bdcba1e46b22bed40a4"
format_stamp: "Formatted at 2026-09-25 10:37:50 on dist-test-slave-dxdd"
server_key: "acd5516eb27caeb947779d8bad527a24"
server_key_iv: "33888134fc19131476e3b35410cfeb08"
server_key_version: "encryptionkey@0"
I20260925 10:37:50.958525 23187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 10:37:50.961941 23202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:50.963218 23187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:37:50.963415 23187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/wal
uuid: "258a4522d3da4bdcba1e46b22bed40a4"
format_stamp: "Formatted at 2026-09-25 10:37:50 on dist-test-slave-dxdd"
server_key: "acd5516eb27caeb947779d8bad527a24"
server_key_iv: "33888134fc19131476e3b35410cfeb08"
server_key_version: "encryptionkey@0"
I20260925 10:37:50.963634 23187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:50.981019 23187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:50.984866 23187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:50.985138 23187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:51.003275 23187 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36507
I20260925 10:37:51.003286 23253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36507 every 8 connection(s)
I20260925 10:37:51.004544 23187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/data/info.pb
I20260925 10:37:51.008001 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:51.012729 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 23187
I20260925 10:37:51.012928 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-0/wal/instance
I20260925 10:37:51.013463 388 external_mini_cluster.cc:1428] Setting key 86ff7b44985684936d5db7a18778500e
I20260925 10:37:51.014907 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:37:51 dist-test-slave-dxdd krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:37:51.039793 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.018649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36507 (local address 127.0.0.1:47888)
0925 10:37:51.019523 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.019578 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:51.022374 (+ 2796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:51.023192 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:51.023202 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:51.023743 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:51.024093 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:51.024110 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.026411 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.026414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:51.027464 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:51.027475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.028702 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.028707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:51.028822 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:51.029473 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:51.029494 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:51.031825 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:51.036019 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:51.036028 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:51.036036 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:51.036507 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:51.036963 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:51.036966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:51.036968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:51.037118 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:51.038063 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:51.038068 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:51.038241 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 10:37:51.038481 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 10:37:51.038803 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":362,"threads_started":1}
I20260925 10:37:51.039829 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.018649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.0.97.62:36507)
0925 10:37:51.020456 (+ 1807us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:51.020462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:51.020502 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:37:51.022733 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:51.022738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:51.022955 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:51.023096 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:51.024307 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.026229 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.027614 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.028503 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.028679 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:51.032057 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:51.032071 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:51.032089 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:51.032131 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:51.035907 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:51.036651 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:51.036661 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:51.036665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:51.036845 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:51.037231 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:51.037238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:51.037242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:51.037930 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:51.038137 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:51.038638 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 10:37:51.038977 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":1633,"spinlock_wait_cycles":15744,"thread_start_us":1400,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260925 10:37:51.042500 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260925 10:37:51.043439 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.043054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40753 (local address 127.0.0.1:37148)
0925 10:37:51.043198 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.043329 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:37:51.043643 23203 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40753: connect: Connection refused (error 111)
I20260925 10:37:51.045220 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.044859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40753 (local address 127.0.0.1:37162)
0925 10:37:51.044981 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.045064 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 10:37:51.045538 23254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40753: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40753: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260925 10:37:51.075191 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40753
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36507,127.0.97.61:40753,127.0.97.60:33847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:37:51.094921 23254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } attempt: 1
I20260925 10:37:51.096238 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.095665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40753 (local address 127.0.0.1:37176)
0925 10:37:51.095908 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.096027 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 10:37:51.097357 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.096916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40753 (local address 127.0.0.1:37182)
0925 10:37:51.097085 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.097181 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 10:37:51.097635 23254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40753: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40753: connect: Connection refused (error 111)
I20260925 10:37:51.165904 23254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } attempt: 2
I20260925 10:37:51.167403 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.166708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40753 (local address 127.0.0.1:37192)
0925 10:37:51.167032 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.167177 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 10:37:51.168659 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.168238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40753 (local address 127.0.0.1:37196)
0925 10:37:51.168406 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.168508 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 10:37:51.168885 23254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40753: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40753: connect: Connection refused (error 111)
I20260925 10:37:51.260186 23254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } attempt: 3
I20260925 10:37:51.261698 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.261111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40753 (local address 127.0.0.1:37202)
0925 10:37:51.261388 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.261516 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 10:37:51.262894 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.262450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40753 (local address 127.0.0.1:37206)
0925 10:37:51.262618 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.262731 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 10:37:51.263108 23254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40753: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40753: connect: Connection refused (error 111)
W20260925 10:37:51.392541 23261 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:37:51.392817 23261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:51.392908 23261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:51.393064 23261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:51.399967 23261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:51.400041 23261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:51.400095 23261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:51.400166 23261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:51.400262 23261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:51.406390 23254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } attempt: 4
I20260925 10:37:51.407557 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.407014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40753 (local address 127.0.0.1:37222)
0925 10:37:51.407290 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.407404 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260925 10:37:51.408481 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.408140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40753 (local address 127.0.0.1:37224)
0925 10:37:51.408273 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.408347 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 10:37:51.408658 23254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40753: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40753: connect: Connection refused (error 111)
I20260925 10:37:51.410358 23261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36507,127.0.97.61:40753,127.0.97.60:33847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:51.411473 23261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:51.412653 23261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:37:51.424219 23266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:51.424275 23269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:51.424309 23267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:51.425093 23261 server_base.cc:1034] running on GCE node
I20260925 10:37:51.425733 23261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:51.427062 23261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:51.428289 23261 hybrid_clock.cc:648] HybridClock initialized: now 1790332671428227 us; error 36 us; skew 500 ppm
I20260925 10:37:51.428545 23261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:51 dist-test-slave-dxdd krb5kdc[23174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:51.431497 23261 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260925 10:37:51.432808 23261 webserver.cc:460] Webserver started at http://127.0.97.61:44883/ using document root <none> and password file <none>
I20260925 10:37:51.433418 23261 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:51.433499 23261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:51.433743 23261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:51.436504 23261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/data/instance:
uuid: "451b7e4b42b44012b228966be823cad9"
format_stamp: "Formatted at 2026-09-25 10:37:51 on dist-test-slave-dxdd"
server_key: "2620be577a5c9369a792844a9d285951"
server_key_iv: "10f63d1da9a2a4742d9ce9e22e710a98"
server_key_version: "encryptionkey@0"
I20260925 10:37:51.437182 23261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/wal/instance:
uuid: "451b7e4b42b44012b228966be823cad9"
format_stamp: "Formatted at 2026-09-25 10:37:51 on dist-test-slave-dxdd"
server_key: "2620be577a5c9369a792844a9d285951"
server_key_iv: "10f63d1da9a2a4742d9ce9e22e710a98"
server_key_version: "encryptionkey@0"
I20260925 10:37:51.441851 23261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 10:37:51.445144 23276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:51.446542 23261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 10:37:51.446668 23261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/wal
uuid: "451b7e4b42b44012b228966be823cad9"
format_stamp: "Formatted at 2026-09-25 10:37:51 on dist-test-slave-dxdd"
server_key: "2620be577a5c9369a792844a9d285951"
server_key_iv: "10f63d1da9a2a4742d9ce9e22e710a98"
server_key_version: "encryptionkey@0"
I20260925 10:37:51.446830 23261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:51.461897 23261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:51.465467 23261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:51.465664 23261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:51.482568 23261 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40753
I20260925 10:37:51.482575 23327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40753 every 8 connection(s)
I20260925 10:37:51.483886 23261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/data/info.pb
I20260925 10:37:51.487469 23328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:51.487891 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 23261
I20260925 10:37:51.488075 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-1/wal/instance
I20260925 10:37:51.488552 388 external_mini_cluster.cc:1428] Setting key 0c0a947d5076b9438db8ae60b702737b
I20260925 10:37:51.493799 23328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:37:51 dist-test-slave-dxdd krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260925 10:37:51.517048 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.496446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.0.97.62:36507)
0925 10:37:51.496619 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:51.496623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:51.497862 (+ 1239us) server_negotiation.cc:407] Connection header received
0925 10:37:51.500691 (+ 2829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:51.500696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:51.500763 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:51.500841 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:51.502868 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.503528 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.506196 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.506687 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.506785 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:51.510949 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:51.510953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:51.510956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:51.510987 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:51.513616 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:51.514372 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:51.514376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:51.514378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:51.514566 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:51.514983 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:51.514988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:51.514992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:51.515310 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:51.515495 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:51.516415 (+ 920us) server_negotiation.cc:299] Negotiation successful
0925 10:37:51.516668 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:37:51.517831 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.496910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36507 (local address 127.0.0.1:38444)
0925 10:37:51.497670 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.497715 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:51.500412 (+ 2697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:51.501082 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:51.501100 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:51.501912 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:51.502697 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:51.502723 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.503684 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.503693 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:51.505926 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:51.505943 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.506920 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.506929 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:51.507245 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:51.507996 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:51.508028 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:51.510721 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:51.513754 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:51.513762 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:51.513779 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:51.514204 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:51.514690 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:51.514695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:51.514699 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:51.514866 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:51.515465 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:51.515473 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:51.515881 (+ 408us) client_negotiation.cc:769] Sending connection context
0925 10:37:51.516300 (+ 419us) client_negotiation.cc:240] Negotiation successful
0925 10:37:51.516729 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":280,"threads_started":1}
I20260925 10:37:51.519646 23328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:37:51 dist-test-slave-dxdd krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260925 10:37:51.533919 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.520214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40753 (local address 127.0.0.1:56032)
0925 10:37:51.520412 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.520431 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:51.520524 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:51.521589 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:51.521593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:51.521936 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:51.522065 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:51.522072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.523993 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.523997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:51.524826 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:51.524835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.525608 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.525613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:51.525798 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:51.526385 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:51.526407 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:51.528423 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:51.531545 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:51.531552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:51.531557 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:51.531895 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:51.532260 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:51.532264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:51.532268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:51.532390 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:51.532967 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:51.532977 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:51.533117 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 10:37:51.533248 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 10:37:51.533533 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 10:37:51.534005 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.520494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.0.97.61:40753)
0925 10:37:51.521088 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:51.521099 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:51.521131 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:37:51.521188 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:51.521198 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:51.521435 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:51.521582 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:51.522236 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.523853 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.524964 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.525476 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.525604 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:51.528637 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:51.528646 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:51.528677 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:51.528707 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:51.531410 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:51.532016 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:51.532024 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:51.532027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:51.532136 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:51.532496 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:51.532503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:51.532507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:51.532840 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:51.533054 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:51.533398 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 10:37:51.533646 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":407,"thread_start_us":263,"threads_started":1}
I20260925 10:37:51.536458 23328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:37:51.537668 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.537237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53500)
0925 10:37:51.537406 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.537525 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 10:37:51.538089 23278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
I20260925 10:37:51.539613 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.539317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53504)
0925 10:37:51.539449 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.539521 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 10:37:51.539911 23328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33847: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260925 10:37:51.549119 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ieADW/build/asan/bin/kudu
/tmp/dist-test-task4ieADW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33847
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36507,127.0.97.61:40753,127.0.97.60:33847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:37:51.589289 23328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } attempt: 1
I20260925 10:37:51.590503 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.590019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53518)
0925 10:37:51.590204 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.590291 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 10:37:51.591439 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.591096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53530)
0925 10:37:51.591261 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.591331 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:37:51.591635 23328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33847: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
I20260925 10:37:51.659845 23328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } attempt: 2
I20260925 10:37:51.660985 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.660517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53538)
0925 10:37:51.660735 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.660840 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 10:37:51.662099 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.661659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53544)
0925 10:37:51.661861 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.661952 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260925 10:37:51.662405 23328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33847: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
I20260925 10:37:51.707939 23254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } attempt: 5
Sep 25 10:37:51 dist-test-slave-dxdd krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260925 10:37:51.723029 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.708970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.0.97.61:40753)
0925 10:37:51.709140 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:51.709144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:51.709163 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:37:51.709334 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:51.709338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:51.709399 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:51.709484 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:51.710591 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.711329 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.713041 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.713463 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.713559 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:51.717170 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:51.717175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:51.717179 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:51.717222 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:51.720125 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:51.721005 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:51.721010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:51.721012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:51.721130 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:51.721660 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:51.721664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:51.721666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:51.721958 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:51.722077 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:51.722633 (+ 556us) server_negotiation.cc:299] Negotiation successful
0925 10:37:51.722798 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:37:51.723275 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.708792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40753 (local address 127.0.0.1:56034)
0925 10:37:51.709009 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.709033 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:51.709174 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:51.709565 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:51.709572 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:51.710104 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:51.710417 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:51.710433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.711483 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.711488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:51.712858 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:51.712873 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.713610 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.713616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:51.713795 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:51.714457 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:51.714479 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:51.716943 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:51.720285 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:51.720293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:51.720300 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:51.720877 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:51.721331 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:51.721339 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:51.721344 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:51.721536 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:51.722096 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:51.722103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:51.722295 (+ 192us) client_negotiation.cc:769] Sending connection context
0925 10:37:51.722522 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 10:37:51.722832 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 10:37:51.724392 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:37:51.725508 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.725115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53558)
0925 10:37:51.725301 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.725376 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 10:37:51.726513 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.726129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53560)
0925 10:37:51.726273 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.726401 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:37:51.726756 23254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33847: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
I20260925 10:37:51.753651 23328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } attempt: 3
I20260925 10:37:51.754841 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.754225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53570)
0925 10:37:51.754500 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.754624 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 10:37:51.756062 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.755518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53586)
0925 10:37:51.755811 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.755915 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260925 10:37:51.756331 23328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33847: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
I20260925 10:37:51.777994 23254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } attempt: 1
I20260925 10:37:51.779335 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.778873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53598)
0925 10:37:51.779070 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.779172 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 10:37:51.780395 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.780070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53606)
0925 10:37:51.780209 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.780287 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 10:37:51.780639 23254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33847: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
I20260925 10:37:51.848891 23254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } attempt: 2
I20260925 10:37:51.850396 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.849626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53612)
0925 10:37:51.849982 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.850099 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 10:37:51.851644 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.851165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53618)
0925 10:37:51.851394 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.851486 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260925 10:37:51.851951 23254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33847: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
W20260925 10:37:51.879590 23335 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:37:51.879949 23335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:37:51.880076 23335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:37:51.880221 23335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:37:51.887447 23335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:37:51.887557 23335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:37:51.887687 23335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:37:51.887775 23335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:37:51.887884 23335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:37:51.897514 23335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36507,127.0.97.61:40753,127.0.97.60:33847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:37:51.898866 23335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:37:51.899581 23328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } attempt: 4
I20260925 10:37:51.900381 23335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:37:51.900743 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.900185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53622)
0925 10:37:51.900452 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.900552 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 10:37:51.901777 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.901418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53638)
0925 10:37:51.901560 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.901639 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:37:51.902086 23328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33847: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
W20260925 10:37:51.912338 23340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:51.912295 23341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:37:51.912389 23343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:37:51.913100 23335 server_base.cc:1034] running on GCE node
I20260925 10:37:51.913745 23335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:37:51.915012 23335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:37:51.916267 23335 hybrid_clock.cc:648] HybridClock initialized: now 1790332671916214 us; error 48 us; skew 500 ppm
I20260925 10:37:51.916584 23335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:37:51 dist-test-slave-dxdd krb5kdc[23174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:37:51.919637 23335 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260925 10:37:51.921020 23335 webserver.cc:460] Webserver started at http://127.0.97.60:35571/ using document root <none> and password file <none>
I20260925 10:37:51.921681 23335 fs_manager.cc:359] Metadata directory not provided
I20260925 10:37:51.921813 23335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:37:51.922115 23335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:37:51.924978 23335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/data/instance:
uuid: "5327dc6d8638481faec54682f03a8a92"
format_stamp: "Formatted at 2026-09-25 10:37:51 on dist-test-slave-dxdd"
server_key: "e0b80636a26d29a23be7afff61e0247c"
server_key_iv: "d79b9e5fc2b9a5143eaeb131c486b30c"
server_key_version: "encryptionkey@0"
I20260925 10:37:51.925750 23335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/wal/instance:
uuid: "5327dc6d8638481faec54682f03a8a92"
format_stamp: "Formatted at 2026-09-25 10:37:51 on dist-test-slave-dxdd"
server_key: "e0b80636a26d29a23be7afff61e0247c"
server_key_iv: "d79b9e5fc2b9a5143eaeb131c486b30c"
server_key_version: "encryptionkey@0"
I20260925 10:37:51.930542 23335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 10:37:51.933723 23350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:37:51.934993 23335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:37:51.935168 23335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/data,/tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/wal
uuid: "5327dc6d8638481faec54682f03a8a92"
format_stamp: "Formatted at 2026-09-25 10:37:51 on dist-test-slave-dxdd"
server_key: "e0b80636a26d29a23be7afff61e0247c"
server_key_iv: "d79b9e5fc2b9a5143eaeb131c486b30c"
server_key_version: "encryptionkey@0"
I20260925 10:37:51.935390 23335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:37:51.958187 23254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } attempt: 3
I20260925 10:37:51.958866 23335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:37:51.959282 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.958799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53646)
0925 10:37:51.959037 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.959138 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 10:37:51.960387 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.959970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53656)
0925 10:37:51.960174 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.960251 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260925 10:37:51.960593 23254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33847: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33847: connect: Connection refused (error 111)
I20260925 10:37:51.963446 23335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:37:51.963701 23335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:37:51.980571 23335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33847
I20260925 10:37:51.980593 23401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33847 every 8 connection(s)
I20260925 10:37:51.981848 23335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/data/info.pb
I20260925 10:37:51.984366 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ieADW/build/asan/bin/kudu as pid 23335
I20260925 10:37:51.984548 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ieADW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332315635355-388-0/minicluster-data/master-2/wal/instance
I20260925 10:37:51.984983 388 external_mini_cluster.cc:1428] Setting key ca922c1c8847038811cd85d54bca0e56
I20260925 10:37:51.985596 23402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:37:51.994904 23402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:37:51 dist-test-slave-dxdd krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 25 10:37:52 dist-test-slave-dxdd krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260925 10:37:52.007184 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.986669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.0.97.60:33847)
0925 10:37:51.987469 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:51.987483 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:51.987529 (+ 46us) server_negotiation.cc:407] Connection header received
0925 10:37:51.988997 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:51.989016 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:51.989403 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:51.989733 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:51.991477 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.992760 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.994436 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:51.995045 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:51.995182 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:51.999431 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:51.999439 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:51.999448 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:51.999481 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:52.002527 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.003409 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.003417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.003424 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.003643 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.004126 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.004131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.004134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.004830 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:52.005150 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:52.005515 (+ 365us) server_negotiation.cc:299] Negotiation successful
0925 10:37:52.005926 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":363,"threads_started":1}
I20260925 10:37:52.013583 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.997503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.0.97.62:36507)
0925 10:37:51.997757 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:51.997767 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:51.998503 (+ 736us) server_negotiation.cc:407] Connection header received
0925 10:37:51.998688 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:51.998695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:51.998790 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:51.998904 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:51.999983 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.001052 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.002535 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.003074 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.003222 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:52.006993 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:52.006999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:52.007006 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:52.007048 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:52.010732 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.011603 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.011608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.011612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.011750 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.012119 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.012124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.012127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.012376 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:52.012547 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:52.013137 (+ 590us) server_negotiation.cc:299] Negotiation successful
0925 10:37:52.013328 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:37:52.013630 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:51.997458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36507 (local address 127.0.0.1:38458)
0925 10:37:51.998348 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:51.998400 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:51.998556 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:51.998959 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:51.998963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:51.999548 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:51.999820 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:51.999829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.001252 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.001256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:52.002405 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:52.002416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.003251 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.003257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:52.003462 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:52.004084 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:52.004111 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:52.006791 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:52.010859 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:52.010868 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:52.010874 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:52.011458 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:52.011878 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:52.011887 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:52.011889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:52.012003 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:52.012517 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:52.012524 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:52.012748 (+ 224us) client_negotiation.cc:769] Sending connection context
0925 10:37:52.012957 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 10:37:52.013250 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":515,"threads_started":1}
I20260925 10:37:52.015280 23402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:37:52 dist-test-slave-dxdd krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 25 10:37:52 dist-test-slave-dxdd krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260925 10:37:52.025435 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:52.010130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.0.97.62:36507)
0925 10:37:52.010755 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.010759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.011060 (+ 301us) server_negotiation.cc:407] Connection header received
0925 10:37:52.011205 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.011210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.011288 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.011401 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.012773 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.013971 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.015600 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.016122 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.016280 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:52.020222 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:52.020227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:52.020230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:52.020258 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:52.022914 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.023528 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.023535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.023545 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.023659 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.024042 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.024047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.024050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.024308 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:52.024573 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:52.024858 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 10:37:52.025077 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":348,"threads_started":1}
I20260925 10:37:52.027832 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:52.015906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40753 (local address 127.0.0.1:56046)
0925 10:37:52.016035 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:52.016058 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:52.016168 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:52.016532 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:52.016536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:52.016887 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:52.017012 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:52.017020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.017942 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.017947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:52.018831 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:52.018842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.019622 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.019627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:52.019796 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:52.020312 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:52.020328 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:52.022558 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:52.025570 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:52.025575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:52.025579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:52.025915 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:52.026301 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:52.026330 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:52.026334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:52.026478 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:52.026960 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:52.026964 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:52.027107 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 10:37:52.027235 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 10:37:52.027508 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 10:37:52.027969 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:52.016059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.0.97.61:40753)
0925 10:37:52.016278 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.016284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.016299 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:37:52.016347 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.016350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.016412 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.016517 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.017160 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.017795 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.018994 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.019480 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.019611 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:52.022736 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:52.022740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:52.022743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:52.022775 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:52.025442 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.026037 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.026043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.026046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.026193 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.026584 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.026590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.026594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.026841 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:52.026997 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:52.027371 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 10:37:52.027590 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:37:52.028699 23402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:37:52 dist-test-slave-dxdd krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 25 10:37:52 dist-test-slave-dxdd krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260925 10:37:52.042229 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:52.029364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.0.97.60:33847)
0925 10:37:52.029522 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.029530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.029722 (+ 192us) server_negotiation.cc:407] Connection header received
0925 10:37:52.029848 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.029851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.029907 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.029974 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.030795 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.031858 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.033352 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.033974 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.034093 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:52.036999 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:52.037003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:52.037006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:52.037032 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:52.039612 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.040267 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.040272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.040275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.040446 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.040851 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.040854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.040857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.041105 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:52.041247 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:52.041580 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 10:37:52.041794 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":52}
I20260925 10:37:52.042292 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:52.029361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53678)
0925 10:37:52.029630 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:52.029647 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:52.029756 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:52.030037 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:52.030042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:52.030499 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:52.030653 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:52.030664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.031983 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.031986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:52.033211 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:52.033223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.034118 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.034122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:52.034292 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:52.034842 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:52.034859 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:52.036856 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:52.039750 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:52.039756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:52.039760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:52.040154 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:52.040578 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:52.040583 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:52.040585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:52.040745 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:52.041220 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:52.041223 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:52.041328 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 10:37:52.041504 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 10:37:52.041932 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":186}
I20260925 10:37:52.043246 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:52.027063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56050 (local address 127.0.97.61:40753)
0925 10:37:52.027708 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.027713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.027735 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:37:52.027813 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.027818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.027914 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.028023 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.029555 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.030717 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.032310 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.032968 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.033092 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:52.037668 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:52.037676 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:52.037681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:52.037721 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:52.040504 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.041150 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.041156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.041161 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.041305 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.041730 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.041735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.041739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.042073 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:52.042471 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:52.042686 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 10:37:52.042827 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":316,"threads_started":1}
I20260925 10:37:52.047789 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:37:52.048358 23402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92: Bootstrap starting.
I20260925 10:37:52.051654 23402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:52.052620 23402 log.cc:824] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:52.049994 23229 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "258a4522d3da4bdcba1e46b22bed40a4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38460
I20260925 10:37:52.055259 23402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92: No bootstrap required, opened a new log
I20260925 10:37:52.061923 23402 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } }
I20260925 10:37:52.062237 23402 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5327dc6d8638481faec54682f03a8a92, State: Initialized, Role: FOLLOWER
I20260925 10:37:52.062747 23402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } }
I20260925 10:37:52.063570 23410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } } }
I20260925 10:37:52.063870 23410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:37:52.064553 23402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 10:37:52.072666 23421 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:37:52.072798 23421 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:37:52.071309 23335 master_runner.cc:186] Error getting master registration for 127.0.97.62:36507: OK, instance_id { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" instance_seqno: 1790332670998086 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:37:52.082412 23335 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:37:52.137831 23254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } attempt: 4
Sep 25 10:37:52 dist-test-slave-dxdd krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260925 10:37:52.151981 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:52.138672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53682)
0925 10:37:52.138872 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:52.138894 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:52.139043 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:52.139466 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:52.139471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:52.139976 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:52.140173 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:52.140182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.141186 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.141195 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:52.142140 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:52.142151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.142955 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.142961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:52.143149 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:52.143785 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:52.143802 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:52.145840 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:52.149223 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:52.149230 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:52.149234 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:52.149572 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:52.150030 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:52.150035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:52.150039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:52.150225 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:52.150997 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:52.151004 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:52.151154 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 10:37:52.151293 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 10:37:52.151588 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 10:37:52.151986 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:52.138946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.0.97.60:33847)
0925 10:37:52.139164 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.139170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.139187 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:37:52.139251 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.139255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.139314 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.139433 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.140324 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.141049 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.142279 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.142808 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.142943 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:52.146026 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:52.146031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:52.146034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:52.146064 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:52.149098 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.149690 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.149695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.149699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.149830 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.150359 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.150364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.150367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.150861 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:52.151044 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:52.151414 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 10:37:52.151601 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:37:52.157907 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4: Bootstrap starting.
I20260925 10:37:52.162994 23254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:52.164428 23254 log.cc:824] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:52.173848 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4: No bootstrap required, opened a new log
I20260925 10:37:52.181253 23254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } }
I20260925 10:37:52.181646 23254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 258a4522d3da4bdcba1e46b22bed40a4, State: Initialized, Role: FOLLOWER
I20260925 10:37:52.182120 23254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } }
I20260925 10:37:52.183352 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } } }
I20260925 10:37:52.184041 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:37:52.184581 23254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:52.184674 23229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 10:37:52.184854 23229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:52.188783 23229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } }
I20260925 10:37:52.191782 23377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "258a4522d3da4bdcba1e46b22bed40a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5327dc6d8638481faec54682f03a8a92"
I20260925 10:37:52.192325 23377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:52.196978 23377 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 258a4522d3da4bdcba1e46b22bed40a4 in term 1.
I20260925 10:37:52.197855 23204 leader_election.cc:302] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 258a4522d3da4bdcba1e46b22bed40a4, 5327dc6d8638481faec54682f03a8a92; no voters:
I20260925 10:37:52.198513 23229 leader_election.cc:288] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [CANDIDATE]: Term 1 election: Requested vote from peers 451b7e4b42b44012b228966be823cad9 (127.0.97.61:40753), 5327dc6d8638481faec54682f03a8a92 (127.0.97.60:33847)
I20260925 10:37:52.203110 23328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } attempt: 5
Sep 25 10:37:52 dist-test-slave-dxdd krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260925 10:37:52.215662 23356 tablet.cc:2369] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:37:52.217080 23424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:37:52.220732 23424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [term 1 LEADER]: Becoming Leader. State: Replica: 258a4522d3da4bdcba1e46b22bed40a4, State: Running, Role: LEADER
I20260925 10:37:52.221796 23424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } }
I20260925 10:37:52.229621 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:52.203908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.0.97.60:33847)
0925 10:37:52.204102 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.204108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.204476 (+ 368us) server_negotiation.cc:407] Connection header received
0925 10:37:52.204554 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.204591 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.204706 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.204821 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.206372 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.207827 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.210238 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.211130 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.211289 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:52.216848 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:52.216889 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:52.216895 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:52.216962 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:52.222928 (+ 5966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.224228 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.224234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.224238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.224496 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.225218 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.225223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.225227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.225936 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:52.226826 (+ 890us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:52.227233 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 10:37:52.228030 (+ 797us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:37:52.230021 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:52.203808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33847 (local address 127.0.0.1:53696)
0925 10:37:52.204187 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 10:37:52.204213 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:37:52.204353 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:37:52.205173 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:37:52.205181 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:37:52.205881 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:37:52.206159 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:52.206175 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.208036 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.208045 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:52.210025 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:37:52.210044 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.211568 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.211577 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:37:52.211892 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:52.212679 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:37:52.212713 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:37:52.216492 (+ 3779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:37:52.223136 (+ 6644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:52.223143 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:52.223149 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:52.224051 (+ 902us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:52.224727 (+ 676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:37:52.224733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:37:52.224739 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:37:52.225051 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:37:52.226149 (+ 1098us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:37:52.226157 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:37:52.226506 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 10:37:52.227183 (+ 677us) client_negotiation.cc:240] Negotiation successful
0925 10:37:52.228239 (+ 1056us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260925 10:37:52.231640 23430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 258a4522d3da4bdcba1e46b22bed40a4. Latest consensus state: current_term: 1 leader_uuid: "258a4522d3da4bdcba1e46b22bed40a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } } }
I20260925 10:37:52.232152 23430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:52.236768 23328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9: Bootstrap starting.
W20260925 10:37:52.237829 23208 tablet.cc:2369] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:37:52.240608 23436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:37:52.242453 23328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9: Neither blocks nor log segments found. Creating new log.
I20260925 10:37:52.244294 23328 log.cc:824] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:37:52.263934 23328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9: No bootstrap required, opened a new log
I20260925 10:37:52.274156 23436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:37:52.293099 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:52.253354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53704 (local address 127.0.97.60:33847)
0925 10:37:52.253556 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.253562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.258975 (+ 5413us) server_negotiation.cc:407] Connection header received
0925 10:37:52.264738 (+ 5763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.264747 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.264862 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.264935 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.267583 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.268712 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.278356 (+ 9644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.279108 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.279208 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:52.283556 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:52.283563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:52.283568 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:52.283608 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:52.287236 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.288226 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.288235 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.288241 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.288435 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.289998 (+ 1563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.290002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.290005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.290386 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:52.290689 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:52.292334 (+ 1645us) server_negotiation.cc:299] Negotiation successful
0925 10:37:52.292633 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:37:52.295281 23328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } }
I20260925 10:37:52.296700 23328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451b7e4b42b44012b228966be823cad9, State: Initialized, Role: FOLLOWER
I20260925 10:37:52.298720 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:52.246651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.0.97.62:36507)
0925 10:37:52.246903 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.246912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.247754 (+ 842us) server_negotiation.cc:407] Connection header received
0925 10:37:52.248211 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.248220 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.248322 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.248426 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.253938 (+ 5512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.255298 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.263372 (+ 8074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.264120 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.264242 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:52.277751 (+ 13509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:52.277758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:52.277763 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:52.277798 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:52.281496 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.291536 (+ 10040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.291544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.291549 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.291713 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.295922 (+ 4209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.295928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.295933 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.296324 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:52.296600 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:52.298083 (+ 1483us) server_negotiation.cc:299] Negotiation successful
0925 10:37:52.298338 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 10:37:52.298272 23328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } }
I20260925 10:37:52.312041 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:52.248247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56056 (local address 127.0.97.61:40753)
0925 10:37:52.248429 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.248434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.255967 (+ 7533us) server_negotiation.cc:407] Connection header received
0925 10:37:52.256254 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.256260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.256333 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.256410 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.266326 (+ 9916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.267389 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.272781 (+ 5392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:52.273428 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:52.273539 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:52.299929 (+ 26390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:37:52.299936 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:37:52.299941 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:37:52.299983 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:37:52.303699 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.308449 (+ 4750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.308455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.308458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.308623 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:37:52.309428 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:37:52.309434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:37:52.309437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:37:52.309814 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:37:52.310061 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:52.311458 (+ 1397us) server_negotiation.cc:299] Negotiation successful
0925 10:37:52.311681 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:37:52.312610 23377 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92 [term 1 FOLLOWER]: Refusing update from remote peer 258a4522d3da4bdcba1e46b22bed40a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:37:52.315354 23430 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:37:52.331501 23303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "258a4522d3da4bdcba1e46b22bed40a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "451b7e4b42b44012b228966be823cad9"
I20260925 10:37:52.331939 23302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:37:52.332020 23328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:37:52.337143 23302 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9 [term 1 FOLLOWER]: Refusing update from remote peer 258a4522d3da4bdcba1e46b22bed40a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:37:52.341702 23303 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 258a4522d3da4bdcba1e46b22bed40a4 in term 1.
I20260925 10:37:52.344759 23430 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:37:52.330010 23447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } } }
I20260925 10:37:52.353791 23447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:37:52.377228 23410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 258a4522d3da4bdcba1e46b22bed40a4. Latest consensus state: current_term: 1 leader_uuid: "258a4522d3da4bdcba1e46b22bed40a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } } }
I20260925 10:37:52.377710 23410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:37:52.382489 23430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "258a4522d3da4bdcba1e46b22bed40a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } } }
I20260925 10:37:52.382750 23430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:52.383860 23436 catalog_manager.cc:1347] Generated new cluster ID: 66985a29604441bd91b1ede4de1a7328
I20260925 10:37:52.383975 23436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:37:52.392354 23410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "258a4522d3da4bdcba1e46b22bed40a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } } }
I20260925 10:37:52.392591 23410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:37:52.393029 23449 mvcc.cc:205] Tried to move back new op lower bound from 7333202625768685568 to 7333202625438679040. Current Snapshot: MvccSnapshot[applied={T|T < 7333202625768685568}]
I20260925 10:37:52.393738 23447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 258a4522d3da4bdcba1e46b22bed40a4. Latest consensus state: current_term: 1 leader_uuid: "258a4522d3da4bdcba1e46b22bed40a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } } }
I20260925 10:37:52.394157 23447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:37:52.395278 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "258a4522d3da4bdcba1e46b22bed40a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } } }
I20260925 10:37:52.395569 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4 [sys.catalog]: This master's current role is: LEADER
I20260925 10:37:52.391054 23187 master_runner.cc:186] Error getting master registration for 127.0.97.61:40753: OK, instance_id { permanent_uuid: "451b7e4b42b44012b228966be823cad9" instance_seqno: 1790332671477683 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:37:52.410480 23187 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:37:52.413882 23447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "258a4522d3da4bdcba1e46b22bed40a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "258a4522d3da4bdcba1e46b22bed40a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36507 } } peers { permanent_uuid: "451b7e4b42b44012b228966be823cad9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40753 } } peers { permanent_uuid: "5327dc6d8638481faec54682f03a8a92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33847 } } }
I20260925 10:37:52.414157 23447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:37:52.436407 23436 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:37:52.439323 23436 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:37:52.461767 23468 catalog_manager.cc:1259] Loaded cluster ID: 66985a29604441bd91b1ede4de1a7328
I20260925 10:37:52.461963 23468 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9: loading cluster ID for follower catalog manager: success
I20260925 10:37:52.472460 23468 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9: acquiring CA information for follower catalog manager: success
I20260925 10:37:52.475698 23436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 258a4522d3da4bdcba1e46b22bed40a4: Generated new TSK 0
I20260925 10:37:52.476841 23436 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:37:52.478426 23468 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20260925 10:37:52.625840 388 mini_kdc.cc:348] Time spent kdestroy: real 0.133s user 0.000s sys 0.022s
W20260925 10:37:52.687047 23407 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:52.666467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.0.97.62:36507)
0925 10:37:52.666680 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.666686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.685115 (+ 18429us) server_negotiation.cc:407] Connection header received
0925 10:37:52.685354 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.685363 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.685462 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.685570 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.686776 (+ 1206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38486: BlockingRecv error: recv got EOF from 127.0.0.1:38486 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260925 10:37:52.687489 23405 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:52.671115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53716 (local address 127.0.97.60:33847)
0925 10:37:52.671318 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.671326 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.685785 (+ 14459us) server_negotiation.cc:407] Connection header received
0925 10:37:52.685936 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.685942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.686026 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.686137 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.687253 (+ 1116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53716: BlockingRecv error: recv got EOF from 127.0.0.1:53716 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 10:37:52.690102 23408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:52.687934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56062 (local address 127.0.97.61:40753)
0925 10:37:52.688114 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.688119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.688617 (+ 498us) server_negotiation.cc:407] Connection header received
0925 10:37:52.688843 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.688848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.688929 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.688998 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.689839 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56062: BlockingRecv error: recv got EOF from 127.0.0.1:56062 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 10:37:52.719477 23408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:52.717232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.0.97.61:40753)
0925 10:37:52.717410 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.717416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.717941 (+ 525us) server_negotiation.cc:407] Connection header received
0925 10:37:52.718150 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.718155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.718289 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.718462 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.719229 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56074: BlockingRecv error: recv got EOF from 127.0.0.1:56074 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 10:37:52.725715 23408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:52.723744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.0.97.61:40753)
0925 10:37:52.723896 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.723903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.724357 (+ 454us) server_negotiation.cc:407] Connection header received
0925 10:37:52.724547 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.724553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.724635 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.724735 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.725469 (+ 734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56076: BlockingRecv error: recv got EOF from 127.0.0.1:56076 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 10:37:52.736331 23408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:52.731986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.0.97.61:40753)
0925 10:37:52.732131 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.732136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.734921 (+ 2785us) server_negotiation.cc:407] Connection header received
0925 10:37:52.735100 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.735109 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.735211 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.735328 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.736071 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56084: BlockingRecv error: recv got EOF from 127.0.0.1:56084 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 10:37:52.746016 23408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:52.744270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56094 (local address 127.0.97.61:40753)
0925 10:37:52.744448 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.744453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.744708 (+ 255us) server_negotiation.cc:407] Connection header received
0925 10:37:52.744879 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.744884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.744960 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.745062 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.745795 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56094: BlockingRecv error: recv got EOF from 127.0.0.1:56094 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 10:37:52.768776 23408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:52.766596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56100 (local address 127.0.97.61:40753)
0925 10:37:52.766766 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.766773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.767263 (+ 490us) server_negotiation.cc:407] Connection header received
0925 10:37:52.767535 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.767541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.767665 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.767775 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.768543 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56100: BlockingRecv error: recv got EOF from 127.0.0.1:56100 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 10:37:52.791244 23408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:52.788792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56114 (local address 127.0.97.61:40753)
0925 10:37:52.789037 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.789045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.789602 (+ 557us) server_negotiation.cc:407] Connection header received
0925 10:37:52.789800 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.789805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.789916 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.790036 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.790918 (+ 882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56114: BlockingRecv error: recv got EOF from 127.0.0.1:56114 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260925 10:37:52.837215 23408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:52.834206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.0.97.61:40753)
0925 10:37:52.834412 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.834418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.834833 (+ 415us) server_negotiation.cc:407] Connection header received
0925 10:37:52.835060 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.835066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.835154 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.835264 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.836985 (+ 1721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56128: BlockingRecv error: recv got EOF from 127.0.0.1:56128 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 10:37:52.912599 23408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:52.910269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56134 (local address 127.0.97.61:40753)
0925 10:37:52.910481 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:52.910488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:52.911015 (+ 527us) server_negotiation.cc:407] Connection header received
0925 10:37:52.911242 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:52.911249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:52.911384 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:52.911510 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:52.912365 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56134: BlockingRecv error: recv got EOF from 127.0.0.1:56134 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260925 10:37:53.045346 23408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:53.043669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.0.97.61:40753)
0925 10:37:53.043915 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:53.043921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:53.044110 (+ 189us) server_negotiation.cc:407] Connection header received
0925 10:37:53.044264 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:53.044268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:53.044349 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:53.044434 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:53.045089 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56138: BlockingRecv error: recv got EOF from 127.0.0.1:56138 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260925 10:37:53.074615 23421 catalog_manager.cc:1259] Loaded cluster ID: 66985a29604441bd91b1ede4de1a7328
I20260925 10:37:53.074709 23421 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92: loading cluster ID for follower catalog manager: success
I20260925 10:37:53.077061 23421 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92: acquiring CA information for follower catalog manager: success
I20260925 10:37:53.078755 23421 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5327dc6d8638481faec54682f03a8a92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 10:37:53.309595 23408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:37:53.307724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56142 (local address 127.0.97.61:40753)
0925 10:37:53.307936 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:53.307943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:53.308110 (+ 167us) server_negotiation.cc:407] Connection header received
0925 10:37:53.308372 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:53.308381 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:53.308475 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:53.308582 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:37:53.309331 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56142: BlockingRecv error: recv got EOF from 127.0.0.1:56142 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:37:53.480960 23468 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 451b7e4b42b44012b228966be823cad9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:37:53.833073 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:53.827682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.0.97.61:40753)
0925 10:37:53.828195 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:53.828199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:53.828218 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:37:53.828392 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:53.828396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:53.828542 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:53.828662 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:53.829034 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:53.829755 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:53.830998 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:53.831414 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:53.831499 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:53.831900 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:53.832505 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:53.832608 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:53.832695 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 10:37:53.832754 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":260,"threads_started":1}
I20260925 10:37:53.844436 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:53.838898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.0.97.60:33847)
0925 10:37:53.839435 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:53.839440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:53.839458 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:37:53.839654 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:53.839658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:53.839819 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:53.839976 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:53.840392 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:53.841083 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:53.842355 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:53.842748 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:53.842913 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:53.843379 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:53.843850 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:53.844006 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:53.844062 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 10:37:53.844140 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":274,"threads_started":1}
I20260925 10:37:53.857946 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:53.850098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53734 (local address 127.0.97.60:33847)
0925 10:37:53.850338 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:53.850346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:53.851630 (+ 1284us) server_negotiation.cc:407] Connection header received
0925 10:37:53.851903 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:53.851909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:53.851987 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:53.852050 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:53.852855 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:53.853732 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:53.855853 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:53.856519 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:53.856599 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:53.857169 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:53.857291 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:53.857356 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:53.857577 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 10:37:53.857663 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:37:53.858551 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:53.850839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.0.97.61:40753)
0925 10:37:53.851037 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:53.851046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:53.851881 (+ 835us) server_negotiation.cc:407] Connection header received
0925 10:37:53.852098 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:53.852106 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:53.852189 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:53.852283 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:53.853024 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:53.854072 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:53.856389 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:53.856993 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:53.857097 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:53.857737 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:53.857865 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:53.857928 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:53.858184 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 10:37:53.858272 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:37:53.859525 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:37:53.849796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.0.97.62:36507)
0925 10:37:53.852088 (+ 2292us) server_negotiation.cc:206] Beginning negotiation
0925 10:37:53.852095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:37:53.852111 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:37:53.852215 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:37:53.852223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:37:53.852457 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:37:53.852628 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:37:53.853286 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:53.854359 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:53.856557 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:37:53.857185 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:37:53.857314 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:37:53.858021 (+ 707us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:37:53.858622 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:37:53.859131 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0925 10:37:53.859186 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 10:37:53.859270 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2159,"thread_start_us":2069,"threads_started":1}
I20260925 10:37:53.869053 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 23187
I20260925 10:37:53.869935 23249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:53.995726 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 23187
W20260925 10:37:54.015556 23351 connection.cc:535] server connection from 127.0.0.1:53682 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:37:54.016203 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 23261
I20260925 10:37:54.017226 23323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:54.121259 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 23261
I20260925 10:37:54.141419 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4ieADW/build/asan/bin/kudu pid 23335
I20260925 10:37:54.142546 23397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:37:54.244632 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ieADW/build/asan/bin/kudu with pid 23335
2026-09-25T10:37:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4996 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9511 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (358574 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST